[Added 02/24/14: After I got feedback on this post, I realized that it carried unnecessary negative connotations (despite conscious effort on my part to avoid them), and if I were to write it again, I would have framed things differently. See Reflections on a Personal Public Relations Failure: A Lesson in Communication for more information. SIAI (now MIRI) has evolved substantially since 2010 when I wrote this post, and the criticisms made in the post don't apply to MIRI as presently constituted.

Follow-up to: Other Existential Risks, Existential Risk and Public Relations

Related to: Tsuyoku Naritai! (I Want To Become Stronger), Affective Death Spirals, The Proper Use of Doubt, Resist the Happy Death Spiral, The Sin  of Underconfidence

In Other Existential Risks I began my critical analysis of what I understand to be SIAI's most basic claims. In particular I evaluated part of the claim

(1) At the margin, the best way for an organization with SIAI's resources to prevent global existential catastrophe is to promote research on friendly Artificial Intelligence, work against unsafe Artificial Intelligence, and encourage rational thought.

It's become clear to me that before I evaluate the claim

(2) Donating to SIAI is the most cost-effective way for charitable donors to reduce existential risk.

I should (a) articulate my reasons for believing in the importance of self-doubt and (b) give the SIAI staff an opportunity to respond to the points which I raise in the present post as well as my two posts titled Existential Risk and Public Relations and Other Existential Risks.

Yesterday SarahC described to me how she had found Eliezer's post Tsuyoku Naritai! (I Want To Become Stronger) really moving. She explained:

I thought it was good: the notion that you can and must improve yourself, and that you can get farther than you think.

I'm used to the other direction: "humility is the best virtue."

I mean, this is a big fuck-you to the book of Job, and it appeals to me.

I was happy to learn that SarahC had been positively affected by Eliezer's post. Self-actualization is a wonderful thing and it appears as though Eliezer's posting has helped her self-actualize. On the other hand, rereading the post prompted me to notice that there's something about it which I find very problematic. The last few paragraphs of the post read:

Take no pride in your confession that you too are biased; do not glory in your self-awareness of your flaws.  This is akin to the principle of not taking pride in confessing your ignorance; for if your ignorance is a source of pride to you, you may become loathe to relinquish your ignorance when evidence comes knocking.  Likewise with our flaws - we should not gloat over how self-aware we are for confessing them; the occasion for rejoicing is when we have a little less to confess.

Otherwise, when the one comes to us with a plan for correcting the bias, we will snarl, "Do you think to set yourself above us?"  We will shake our heads sadly and say, "You must not be very self-aware."

Never confess to me that you are just as flawed as I am unless you can tell me what you plan to do about it.  Afterward you will still have plenty of flaws left, but that's not the point; the important thing is to do better, to keep moving ahead, to take one more step forward.  Tsuyoku naritai!

There's something to what Eliezer is saying here: when people are too strongly committed to the idea that humans are fallible this can become a self-fulfilling prophecy where humans give up on trying to improve things and as a consequence remain fallible when they could have improved. As Eliezer has said in The Sin of Underconfidence, there are social pressures that push against having high levels of confidence even when confidence is epistemically justified:

To place yourself too high - to overreach your proper place - to think too much of yourself - to put yourself forward - to put down your fellows by implicit comparison - and the consequences of humiliation and being cast down, perhaps publicly - are these not loathesome and fearsome things?

To be too modest - seems lighter by comparison; it wouldn't be so humiliating to be called on it publicly, indeed, finding out that you're better than you imagined might come as a warm surprise; and to put yourself down, and others implicitly above, has a positive tinge of niceness about it, it's the sort of thing that Gandalf would do.

I have personal experience with underconfidence. I'm a careful thinker and when I express a position with confidence my position is typically well considered. For many years I generalized from one example and assumed when people express positions with confidence they've thought their positions out as well as I have. Even after being presented with massive evidence that few people think things through as carefully as I do, I persisted in granting the (statistically ill-considered) positions of others far more weight than they deserved for the very reason that Eliezer describes above. This seriously distorted my epistemology because it led to me systematically giving ill-considered positions substantial weight. I feel that I have improved on this point, but even now, from time to time I notice that I'm exhibiting irrationally low levels of confidence in my positions.

At the same time, I know that at times I've been overconfident as well. In high school I went through a period when I believed that I was a messianic figure whose existence had been preordained by a watchmaker God who planned for me to save the human race. It's appropriate to say that during this period of time I suffered from extreme delusions of grandeur. I viscerally understand how it's possible to fall into an affective death spiral.

In my view one of the central challenges of being human is to find an instrumentally rational balance between subjecting oneself to influences which push one in the direction of overconfidence and subjecting oneself to influences which push one in the direction of underconfidence.

In Tsuyoku Naritai! Eliezer describes how Orthodox Judaism attaches an unhealthy moral significance to humility. Having grown up in a Jewish household and as a consequence having had peripheral acquaintance with orthodox Judaism I agree with Eliezer's analysis of Orthodox Judaism in this regard. In the proper use of doubt, Eliezer describes how the Jesuits allegedly are told to doubt their doubts about Catholicism. I agree with Eliezer that self-doubt can be misguided and abused.

However, reversed stupidity is not intelligence. The fact that it's possible to ascribe too much moral significance to self-doubt and humility does not mean that one should not attach moral significance to self-doubt and humility. I strongly disagree with Eliezer's prescription: "Take no pride in your confession that you too are biased; do not glory in your self-awareness of your flaws."

The mechanism that determines human action is that we do what makes us feel good (at the margin) and refrain from doing what makes us feel bad (at the margin). This principle applies to all humans, from Gandhi to Hilter. Our ethical challenge is to shape what makes us feel good and what makes us feel bad in a way that incentivizes us to behave in accordance with our values. There are times when it's important to recognize that we're biased and flawed. Under such circumstances, we should feel proud that we recognize that we're biased we should glory in our self-awareness of our flaws. If we don't, then we will have no incentive to recognize that we're biased and be aware of our flaws.

We did not evolve to exhibit admirable and noble behavior. We evolved to exhibit behaviors which have historically been correlated with maximizing our reproductive success. Because our ancestral climate was very much a zero-sum situation, the traits that were historically correlated with maximizing our reproductive success had a lot to do with gaining high status within our communities. As Yvain has said, it appears that a fundamental mechanism of the human brain which was historically correlated with gaining high status is to make us feel good when we have high self-image and feel bad when we have low self-image.

When we obtain new data, we fit it into a narrative which makes us feel as good about ourselves as possible; a way conducive to having a high self-image. This mode of cognition can lead to very seriously distorted epistemology. This is what happened to me in high school when I believed that I was a messianic figure sent by a watchmaker God. Because we flatter ourselves by default, it's very important that those of us who aspire to epistemic rationality incorporate a significant element of "I'm the sort of person who engages in self-doubt because it's the right thing to do" into our self-image. If we do this, when we're presented with evidence which entails a drop in our self-esteem, we don't reject it out of hand or minimize it as we've been evolutionarily conditioned to do because wound of properly assimilating data is counterbalanced by the salve of the feeling "At least I'm a good person as evidenced by the fact that I engage in self-doubt" and failing to exhibit self-doubt would itself entail an emotional wound.

This is the only potential immunization to the disease of self-serving narratives which afflicts all utilitarians out of virtue of their being human. Until technology allows us to modify ourselves in a radical way, we cannot hope to be rational without attaching moral significance to the practice of engaging in self-doubt. As the RationalWiki's page on LessWrong says:

A common way for very smart people to be stupid is to think they can think their way out of being apes with pretensions. However, there is no hack that transcends being human...You are an ape with pretensions. Playing a "let's pretend" game otherwise doesn't mean you win all arguments, or any. Even if it's a very elaborate one, you won't transcend being an ape. Any "rationalism" that doesn't expressly take into account humans being apes with pretensions, isn't.


In Existential Risk and Public Relations I suggested that some of Eliezer's remarks convey the impression that Eliezer has an unjustifiably high opinion of himself. In the comments to the post JRMayne wrote

I think the statements that indicate that [Eliezer] is the most important person in human history - and that seems to me to be what he's saying - are so seriously mistaken, and made with such a high confidence level, as to massively reduce my estimated likelihood that SIAI is going to be productive at all.

And that's a good thing. Throwing money into a seriously suboptimal project is a bad idea. SIAI may be good at getting out the word of existential risk (and I do think existential risk is serious, under-discussed business), but the indicators are that it's not going to solve it. I won't give to SIAI if Eliezer stops saying these things, because it appears he'll still be thinking those things.

When Eliezer responded to JRMayne's comment, Eliezer did not dispute the claim that JRMayne attributed to him. I responded to Eliezer saying

If JRMayne has misunderstood you, you can effectively deal with the situation by making a public statement about what you meant to convey.

Note that you have not made a disclaimer which rules out the possibility that you claim that you're the most important person in human history. I encourage you to make such a disclaimer if JRMayne has misunderstood you.

I was disappointed, but not surprised, that Eliezer did not respond. As far as I can tell, Eliezer does have confidence in the idea that he is (at least nearly) the most important person in human history. Eliezer's silence only serves to further confirm my earlier impressions. I hope that Eliezer subsequently proves me wrong. [Edit: As Airedale points out Eliezer has in fact exhibited public self-doubt in his abilities in his posting The Level Above Mine. I find this reassuring and it significantly lowers my confidence that Eliezer claims that he's the most important person in human history. But Eliezer still hasn't made a disclaimer on this matter decisively indicating that he does not hold such a view.] The modern world is sufficiently complicated so that no human no matter how talented can have good reason to believe himself or herself to be the most important person in human history without actually doing something which very visibly and decisively alters the fate of humanity. At present, anybody who holds such a belief is suffering from extreme delusions of grandeur.

There's some sort of serious problem with the present situation. I don't know whether it's a public relations problem or if the situation is that Eliezer actually suffers from extreme delusions of grandeur, but something has gone very wrong. The majority of the people who I know who outside of Less Wrong who have heard of Eliezer and Less Wrong have the impression that Eliezer is suffering from extreme delusions of grandeur. To such people, this fact (quite reasonably) calls into question of the value of SIAI and Less Wrong. On one hand, SIAI looks like an organization which is operating under beliefs which Eliezer has constructed to place himself in as favorable a position as possible rather than with a view toward reducing existential risk. On the other hand, Less Wrong looks suspiciously like the cult of Objectivism: a group of smart people who are obsessed with the writings of a very smart person who is severely deluded and describing these writings and the associated ideology as "rational" although they are nothing of the kind.

My own views are somewhat more moderate. I think that the Less Wrong community and Eliezer are considerably more rational than the Objectivist movement and Ayn Rand (respectively). I nevertheless perceive unsettling parallels.


In the comments to Existential Risk and Public Relations, timtyler said

...many people have inflated views of their own importance. Humans are built that way. For one thing, It helps them get hired, if they claim that they can do the job. It is sometimes funny - but surely not a big deal.

I disagree with timtyler. Anything that has even a slight systematic negative impact on existential risk is a big deal.

Some of my most enjoyable childhood experiences involved playing Squaresoft RPGs. Games like Chrono Trigger, Illusion of Gaia, Earthbound, Xenogears, and the Final Fantasy series are all stories about a group of characters who bond and work together to save the world. I found these games very moving and inspiring. They prompted me to fantasize about meeting allies who I could bond with and work together with to save the world. I was lucky enough to meet one such person in high school who I've been friends with since. When I first encountered Eliezer I found him eerily familiar, as though he was a long lost brother. This is the same feeling that is present between Siegmund and Sieglinde in the Act 1 of Wagner's Die Walküre (modulo erotic connotations). I wish that I could be with Eliezer in a group of characters as in a Squaresoft RPG working to save the world. His writings such as One Life Against the World and Yehuda Yudkowsky, 1985-2004 reveal him to be a deeply humane and compassionate person.

This is why it's so painful for me to observe that Eliezer appears to be deviating so sharply from leading a genuinely utilitarian lifestyle. I feel a sense of mono no aware, wondering how things could have been under different circumstances.

One of my favorite authors is Kazuo Ishiguro, who writes about the themes of self-deception and people's attempts to contribute to society. In a very good interview Ishiguro said

I think that's partly what interests me in people, that we don't just wish to feed and sleep and reproduce then die like cows or sheep. Even if they're gangsters, they seem to want to tell themselves they're good gangsters and they're loyal gangsters, they've fulfilled their 'gangstership' well. We do seem to have this moral sense, however it's applied, whatever we think. We don't seem satisfied, unless we can tell ourselves by some criteria that we have done it well and we haven't wasted it and we've contributed well. So that is one of the things, I think, that distinguishes human beings, as far as I can see.

But so often I've been tracking that instinct we have and actually looking at how difficult it is to fulfill that agenda, because at the same time as being equipped with this kind of instinct, we're not actually equipped. Most of us are not equipped with any vast insight into the world around us. We have a tendency to go with the herd and not be able to see beyond our little patch, and so it is often our fate that we're at the mercy of larger forces that we can't understand. We just do our little thing and hope it works out. So I think a lot of the themes of obligation and so on come from that. This instinct seems to me a kind of a basic thing that's interesting about human beings. The sad thing is that sometimes human beings think they're like that, and they get self-righteous about it, but often, they're not actually contributing to anything they would approve of anyway.

[...]

There is something poignant in that realization: recognizing that an individual's life is very short, and if you mess it up once, that's probably it. But nevertheless, being able to at least take some comfort from the fact that the next generation will benefit from those mistakes. It's that kind of poignancy, that sort of balance between feeling defeated but nevertheless trying to find reason to feel some kind of qualified optimism. That's always the note I like to end on. There are some ways that, as the writer, I think there is something sadly pathetic but also quite noble about this human capacity to dredge up some hope when really it's all over. I mean, it's amazing how people find courage in the most defeated situations.

Ishiguro's quote describes how people often behave in accordance with sincere desire to contribute and end up doing things that are very different from what they thought they were doing (things which are relatively unproductive or even counterproductive). Like Ishiguro I find this phenomenon very sad. As Ishiguro hints at, this phenomenon can also result in crushing disappointment later in life. I feel a deep spiritual desire to prevent this from happening to Eliezer.

New Comment
746 comments, sorted by Click to highlight new comments since:
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

This post suffers from lumping together orthogonal issues and conclusions from them. Let's consider individually the following claims:

  1. The world is in danger, and the feat of saving the world (if achieved) would be very important, more so than most other things we can currently do.
  2. Creating FAI is possible.
  3. Creating FAI, if possible, will be conductive to saving the world.
  4. If FAI is possible, person X's work contributes to developing FAI.
  5. Person X's work contributes to saving the world.
  6. Most people's work doesn't contribute to saving the world.
  7. Person X's activity is more important than that of most other people.
  8. Person X believes their activity is more important than that of most other people.
  9. Person X suffers from delusions of grandeur.

A priori, from (8) we can conclude (9). But assuming the a priori improbable (7), (8) is a rational thing for X to conclude, and (9) doesn't automatically follow. So, at this level of analysis, in deciding whether X is overconfident, we must necessarily evaluate (7). In most cases, (7) is obviously implausible, but the post itself suggests one pattern for recognizing when it isn't:

The modern world is sufficiently complicated so that no human n

... (read more)
7multifoliaterose
Your analysis is very careful and I agree with almost everything that you say. I think that one should be hesitant to claim too much for a single person on account of the issue which Morendil raises - we are all connected. Your ability to work on FAI depends on the farmers who grow your food, the plumbers who ensure that you have access to running water, the teachers who you learned from, the people at Google who make it easier for you to access information, etc. I believe that you (and others working on the FAI problem) can credibly hold the view that your work has higher expected value to humanity than that of a very large majority (e.g. 99.99%) of the population. Maybe higher. I don't believe that Eliezer can credibly hold the view that he's the highest expected value human who has ever lived. Note that he has not offered a disclaimer denying the view that JRMayne has attributed to him despite the fact that I have suggested that he do so twice now.
8Vladimir_Nesov
You wrote elsewhere in the thread: Does it mean that we need 10^9 Eliezer-level researchers to make progress? Considering that Eliezer is probably at about 1 in 10000 level of ability (if we forget about other factors that make research in FAI possible, such as getting in the frame of mind of understanding the problem and taking it seriously), we'd need about 1000 times more human beings than currently exists on the planet to produce a FAI, according to your estimate. How does this claim coexist with the one you've made in the above comment? It doesn't compute, there is an apparent inconsistency between these two claims. (I see some ways to mend it by charitable interpretation, but I'd rather you make the intended meaning explicit yourself.)
3Jonathan_Graehl
Agreed, and I like to imagine that he reads that and thinks to himself "only 10000? thanks a lot!" :) In case anyone takes the above too seriously, I consider it splitting hairs to talk about how much beyond 1 in 10000 smart anyone is - eventually, motivation, luck, and aesthetic sense / rationality begin to dominate in determining results IMO.
1multifoliaterose
No, in general p(n beings similar to A can do X) does not equal n multiplied by p(A can do X). I'll explain my thinking on these matters later.
0Vladimir_Nesov
Yes, strictly speaking we'd need even more, if that. The more serious rendition of my remark is that you seem to imply that the problem itself is not solvable at all, by proxy of the estimate of Eliezer's ability to contribute to the solution. But it's OK, informal conclusions differ; what's not OK is that in the other comment you seem to contradict your claim. Edit: I was not thinking clearly here.
2Tyrrell_McAllister
No. There is a very small chance that I will be able to move my couch down the stairs alone. But it's fairly likely that I and my friend will be able to do it together. Similarly, 10^5 Eliezer-level researchers would together constitute a research community that could do things that Eliezer himself has less than probability 10^(-5) of doing on his own.
3Vladimir_Nesov
Agreed, I was not thinking clearly. The original comment stands, since what you suggest is one way to dissolve the apparent inconsistency, but my elaboration was not lucid.
0multifoliaterose
Tyrrel_MacAllister's remark is a significant part of what I have in mind. I presently think that the benefits of a (modestly) large and diverse research community are very substantial and that SIAI should not attempt to research Friendly AI unilaterally but rather should attempt to collaborate with existing institutions.
9Vladimir_Nesov
I agree about the benefits of larger research community, although feasibility of "collaborating with existing institutions" is in question, due to the extreme difficulty of communicating the problem statement. There are also serious concerns about the end-game, where it will be relatively easy to instantiate a random-preference AGI on the basis of tools developed in the course of researching FAI. Although the instinct is to say "Secrecy in science? Nonsense!", it would also be an example of outside view, where one completes a pattern while ignoring specific detail. Secrecy might make the development of a working theory less feasible, but if open research makes the risks of UFAI correspondingly even worse, it's not what we ought to do. I'm currently ambivalent on this point, but it seems to me that at least preference theory (I'll likely have a post on that on my blog tomorrow) doesn't directly increase the danger, as it's about producing tools sufficient only to define Friendliness (aka human preference), akin to how logic allows to formalize open conjectures in number theory (of course, the definition of Friendliness has to reference some actual human beings, so it won't be simple when taken together with that, unlike conjectures in number theory), with such definition allowing to conclusively represent the correctness of any given (efficient algorithmic) solution, without constructing that solution. On the other hand, I'm not confident that having a definition alone is not sufficient to launch the self-optimization process, given enough time and computing power, and thus published preference theory would constitute a "weapon of math destruction".
1cousin_it
Hey, three days have passed and I want that post!
1Vladimir_Nesov
I have an excuse, I got a cold!
4cousin_it
Okay hurry up then, you're wasting lives in our future light cone.
3wedrifid
"Shut up and do the temporarily inconvenient!"
1cousin_it
Three more days have passed.
1Vladimir_Nesov
Planning is the worst form of procrastination. I now have 7 (!) posts planned before the roadmap post I referred to (with the readmap post closing the sequence), so I decided on writing a mini-sequence of 2-3 posts on LW about ADT first.
1multifoliaterose
Maybe things could gradually change with more interface between people who are interested in FAI and researchers in academia. I agree with this and believe that this could justify secrecy, but I think that it's very important that we hold the people who we trust with the end-game to very high standards for demonstrated epistemic rationality and scrupulousness. I do not believe that the SIAI staff have met such standards. My belief on this matter regard is a major reason why I'm pursuing my current trajectory of postings.
5whpearson
I'd argue that a lot of people's work does. Everybody that contributes to keeping the technological world running (from farmers to chip designers) enables us to potentially save ourselves from the longer term non-anthrogenic existential risks.
6Vladimir_Nesov
Obviously, you need to interpret that statement as "Any given person's work doesn't significantly contribute to saving the world". In other words, if we "subtract" that one person, the future (in the aspect of the world not ending) changes insignificantly.
3whpearson
Are you also amending 4) to have the significant clause? Because there are lots of smart people that have worked on AI, whose work I doubt would be significant. And that is the nearest reference class I have for likely significance of people working on FAI.
1Vladimir_Nesov
I'm not amending, I'm clarifying. (4) doesn't have world-changing power in itself, only through the importance of FAI implied by other arguments, and that part doesn't apply to activity of most people in the world. I consider the work on AI as somewhat significant as well, although obviously less significant than work on FAI at the margain, since much more people are working on AI. The argument, as applied to their work, makes them an existential threat (moderate to high when talking about the whole profession, rather weak when talking about individual people). As for the character of work, I believe that at the current stage, productive work on FAI is close to pure mathematics (but specifically with problem statements not given), and very much unlike most of AI or even the more rigorous kinds from machine learning (statistics).
1MartinB
That makes me wonder who will replace Norman Borlaug, or lets say any particular influential writer or thinker.
2CarlShulman
Agreed. More broadly, everyone affects anthropogenic existential risks too, which limits the number of orders of magnitude one can improve in impact from a positive start.
1Wei Dai
(4 here being "If FAI is possible, person X's work contributes to developing FAI.") This seems be a weak part of your argument. A successful FAI attempt will obviously have to use lots of philosophical and technical results that were not developed specifically with FAI in mind. Many people may be contributing to FAI, without consciously intending to do so. For example when I first started thinking about anthropic reasoning I was mainly thinking about human minds being copyable in the future and trying to solve philosophical puzzles related to that. Another possibility is that the most likely routes to FAI go through intelligence enhancement or uploading, so people working in those fields are actually making more contributions to FAI than people like you and Eliezer.
-1JRMayne
Person X believes that their activity is more important than all other people, and that no other people can do it. Person X also believes that only this project is likely to save the world. Person X also believes that FAI will save the world on all axes, including political and biological. --JRM
-1cata
Generally speaking, your argument isn't very persuasive unless you believe that the world is doomed without FAI and that direct FAI research is the only significant contribution you can make to saving it. (EDIT: To clarify slightly after your response, I mean to point out that you didn't directly mention these particular assumptions, and that I think many people take issue with them.) My personal, rather uninformed belief is that FAI would be a source of enormous good, but it's not necessary for humanity to continue to grow and to overcome x-risk (so 3 is weaker); X may be contributing to the development of FAI, but not that much (so 4 is weaker); and other people engaged in productive pursuits are also contributing a non-zero amount to "save the world" (so 6 is weaker.) As such, I have a hard time concluding that X's activity is anywhere near the "most important" using your reasoning, although it may be quite important.
3Vladimir_Nesov
The argument I gave doesn't include justification of things it assumes (that you referred to). It only serves to separate the issues with claims about a person from issues with claims about what's possible in the world. Both kinds of claims (assumptions in the argument I gave) could be argued with, but necessarily separately.
0cata
OK, I now see what your post was aimed at, a la this other post you made. I agree that criticism ought to be toward person X's beliefs about the world, not his conclusions about himself.

Unknown reminds me that Multifoliaterose said this:

The modern world is sufficiently complicated so that no human no matter how talented can have good reason to believe himself or herself to be the most important person in human history without actually doing something which very visibly and decisively alters the fate of humanity. At present, anybody who holds such a belief is suffering from extreme delusions of grandeur.

This makes explicit something I thought I was going to have to tease out of multi, so my response would roughly go as follows:

  • If no one can occupy this epistemic state, that implies something about the state of the world - i.e., that it should not lead people into this sort of epistemic state.
  • Therefore you are deducing information about the state of the world by arguing about which sorts of thoughts remind you of your youthful delusions of messianity.
  • Reversed stupidity is not intelligence. In general, if you want to know something about how to develop Friendly AI, you have to reason about Friendly AI, rather than reasoning about something else.
  • Which is why I have a policy of keeping my thoughts on Friendly AI to the object level, and not worrying about ho
... (read more)
6Jonathan_Graehl
Upvoted for being clever. You've (probably) refuted the original statement as an absolute. You're deciding not to engage the issue of hubris directly. Does the following paraphrase your position: 1. Here's what I (and also part of SIAI) intend to work on 2. I think it's very important (and you should think so for reasons outline in my writings) 3. If you agree with me, you should support us ? If so, I think it's fine for you to not say the obvious (that you're being quite ambitious, and that success is not assured). It seems like some people are really dying to hear you say the obvious.

Upvoted for being clever.

That's interesting. I downvoted it for being clever. It was a convoluted elaboration of a trivial technicality that only applies if you make the most convenient (for Eliezer) interpretation of multi's words. This kind of response may win someone a debating contest in high school but it certainly isn't what I would expect from someone well versed in the rationalism sequences, much less their author.

I don't pay all that much attention to what multi says (no offence intended to multi) but I pay close attention to what Eliezer does. I am overwhelmingly convinced of Eliezer's cleverness and brilliance as a rationalism theorist. Everything else, well, that's a lot more blurry.

4Furcas
I don't think Eliezer was trying to be clever. He replied to the only real justification multi offered for why we should believe that Eliezer is suffering from delusions of grandeur. What else is he supposed to do?
7wedrifid
I got your reply and respect your position. I don't want to engage too much here since it would overlap with discussion surrounding Eliezer's initial reply and potentially be quite frustrating. What I would like to see is multifoliaterose giving a considered response to the "If not, why not?" question in that link. That would give Eliezer the chance to respond to the meat of the topic at hand. Eliezer has been given a rare opportunity. He can always write posts about himself, giving justifications for whatever degree of personal awesomeness he claims. That's nothing new. But in this situation it wouldn't be perceived as Eliezer grabbing the megaphone for his own self-gratification. He is responding to a challenge, answering a request. Why would you waste the chance to, say, explain the difference between "SIAI" and "Eliezer Yudkowsky"? Or at least give some treatment of p(someone other than Eliezer Yudkowsky is doing the most to save the world). Better yet, take that chance to emphasise the difference between p(FAI is the most important priority for humanity) and p(Eliezer is the most important human in the world).
4khafra
As Graehl and wedrifid observed, Eliezer responded as if the original statement were an absolute. He applied deductive reasoning and found a reductio ad absurdum. But if, instead of an absolute, you see multifoliaterose's characterization as a reference class: "People who believe themselves to be one of the few most important in the world without having already done something visible and obvious to dramatically change it," it can lower the probability that Eliezer is, in fact, that important by a large likelihood ratio. Whether this likelihood ratio is large enough to overcome the evidence on AI-related existential risk and the paucity of serious effort dedicated to combating it is an open question.

Success is not assured. I'm not sure what's meant by confessing to being "ambitious". Is it like being "optimistic"? I suppose there are people who can say "I'm being optimistic" without being aware that they are instantiating Moore's Paradox but I am not one of them.

I also disclaim that I do not believe myself to be the protagonist, because the world is not a story, and does not have a plot.

2Perplexed
I hope that the double negative in the last sentence was an error. I introduced the term "protagonist", because at that point we were discussing a hypothetical person who was being judged regarding his belief in a set of three propositions. Everyone recognized, of course, who that hypothetical person represented, but the actual person had not yet stipulated his belief in that set of propositions.
3wedrifid
Interesting. I don't claim great grammatical expertise but my reading puts the last question at reasonable. Am I correct in inferring that you do not believe Eliezer's usage of "I also disclaim" to mean "I include the following disclaimer: " is valid? Regarding 'protagonist' there is some context for the kind of point Eliezer likes to make about protagonist/story thinking in his Harry Potter fanfic. I don't believe he has expressed the concept coherently as a post yet. (I don't see where you introduced the 'protagonist' word so don't know whether Eliezer read you right. I'm just throwing some background in.)
5Perplexed
Regarding "disclaim". I read "disclaim" as a synonym for "deny". I didn't even consider your interpretation, but upon consideration, I think I prefer it. My mistake (again!). :(
1Vaniver
This question is best solved by a dictionary. "I disclaim that I am a blegg" means that I am not a blegg; "Disclaimer: I am a blegg" means that I am a blegg. The use of disclaimer in the second statement is describing the following statement: "I am making a claim that denies something: I am a blegg." Take home message: Eliezer's double negative means his post has the opposite effect of what I hope he intended.
0Jonathan_Graehl
Yes, that was exactly the sense of "ambitious" I intended - the second person sneering one, which when used by oneself, would be more about signaling humility than truth. I see that's not your style.
3Unknowns
Even if almost everything you say here is right, it wouldn't mean that there is a high probability that if you are killed in a car accident tomorrow, no one else will think about these things (reflective decision theory and so on) in the future, even people who know nothing about you personally. As Carl Shulman points out, if it is necessary to think about these things it is likely that people will, when it becomes more urgent. So it still wouldn't mean that you are the most important person in human history.
2multifoliaterose
I agree with khafra. Your response to my post is distortionary. The statement which you quote was a statement about the reference class of people who believe themselves to be the most important person in the world. The statement which you quote was not a statement about FAI. Any adequate response to the statement which you quote requires that you engage with the last point that khafra made: You have not satisfactorily addressed this matter.
6Furcas
It looks to me like Eliezer gave your post the most generous interpretation possible, i.e. that it actually contained an argument attempting to show that he's deluding himself, rather than just defining a reference class and pointing out that Eliezer fits into it. Since you've now clarified that your post did nothing more than that, there's not much left to do except suggest you read all of Eliezer's posts tagged 'FAI', and this.

give the SIAI staff an opportunity to respond to the points which I raise in the present post as well as my two posts titled Existential Risk and Public Relations and Other Existential Risks.

Indeed, given how busy everyone at SIAI has been with the Summit and the academic workshop following it, it is not surprising that there has not been much response from SIAI. I was only involved as an attendee of the Summit, and even I am only now able to find time to sit down and write something in response. At any rate, as a donor and former visiting fellow, I am only loosely affiliated with SIAI, and my comments here are solely my own, although my thoughts are certainly influenced by observations of the organization and conversation with those at SIAI. I don’t have the time/knowledge to address everything in your posts, but I wanted to say a couple of things.

I don’t disagree with you that SIAI has certain public relations problems. (Frankly, I doubt anyone at SIAI would disagree with that.) There is a lot of attention and discussion at SIAI about how to best spread knowledge about existential risks and to avoid sounding like a fringe/doomsday organization in doing so. It’s true that... (read more)

4Morendil
Speaking from personal experience, the SIAI's somewhat haphazard response to people answering its outreach calls strikes me as a bigger PR problem than Eliezer's personality. The SIAI strikes me as in general not very good at effective collective action (possibly because that's an area where Eliezer's strengths are, as he admits himself, underdeveloped). One thing I'd suggest to correct that is to massively encourage collaborative posts on LW.
3Airedale
Agreed. I think that communication and coordination with many allies and supporters has historically been a weak point for SIAI, due to various reasons including overcommitment of some of those tasked with communications, failure to task anyone with developing or maintaining certain new and ongoing relationships, interpersonal skills being among the less developed skill sets among those at SIAI, and the general growing pains of the organization. My impression is that there has been some improvement in this area recently, but there's still room for a lot more. More collaborative posts on LW would be great to see. There have also been various discussions about workshops or review procedures for top-level posts that seem to have generated at least some interest. Maybe those discussions should just continue in the open thread or maybe it would be appropriate to have a top-level post where people could be invited to volunteer or could find others interested in collaboration, workshops, or the like.
1multifoliaterose
Thanks for pointing out "The Level Above Mine." I had not seen it before.

I'd like to vote this up as I agree with lots of the points raised, but I am not comfortable with the personal nature of this article. I'd much rather the bits personal to Eliezer be sent via email.

Probably some strange drama avoidance thing on my part. On the other hand I'm not sure Eliezer would have a problem writing a piece like this about someone else.

I've thought to myself that I have read one too many fantasy books as a kid, so the partying metaphor hits home.

I'd like to vote this up as I agree with lots of the points raised, but I am not comfortable with the personal nature of this article. I'd much rather the bits personal to Eliezer be sent via email.

I was conflicted about posting in the way that I did precisely for the reason that you describe, but after careful consideration decided that the benefits outweighed the costs, in part because Eliezer does not appear to be reading the private messages that I send him.

4JamesAndrix
I would say that given an audience that is mostly not Eliezer. the best way to send a personal message to Eliezer is to address how the community ought to relate to Eliezer.
0multifoliaterose
Yes, what you say makes sense. If I were to write my post again I would have framed the issues therein somewhat differently.
[-]ata200

I'm inclined to think that Eliezer's clear confidence in his own very high intelligence and his apparent high estimation of his expected importance (not the dictionary-definition "expected", but rather, measured as an expected quantity the usual way) are not actually unwarranted, and only violate the social taboo against admitting to thinking highly of one's own intelligence and potential impact on the world, but I hope he does take away from this a greater sense of the importance of a "the customer is always right" attitude in managing his image as a public-ish figure. Obviously the customer is not always right, but sometimes you have to act like they are if you want to get/keep them as your customer... justified or not, there seems to be something about this whole endeavour (including but not limited to Eliezer's writings) that makes people think !!!CRAZY!!! and !!!DOOMSDAY CULT!!!, and even if is really they who are the crazy ones, they are nevertheless the people who populate this crazy world we're trying to fix, and the solution can't always just be "read the sequences until you're rational enough to see why this makes sense".

I realize it's a bala... (read more)

there seems to be something about this whole endeavour (including but not limited to Eliezer's writings) that makes people think !!!CRAZY!!! and !!!DOOMSDAY CULT!!!,

Yes, and it's called "pattern completion", the same effect that makes people think "Singularitarians believe that only people who believe in the Singularity will be saved".

2Emile
This is discussed in Imaginary Positions.
0ata
I must know, have you actually encountered people who literally think that? I'm really hoping that's a comical exaggeration, but I guess I should not overestimate human brains.
7timtyler
"It's basically a modern version of a religious belief system and there's no purpose to it, like why, why must we have another one of these things ... you get an afterlife out of it because you'll be on the inside track when the singularity happens - it's got all the trappings of a religion, it's the same thing." - Jaron here.
7Eliezer Yudkowsky
I've encountered people who think Singularitarians think that, never any actual Singularitarians who think that.
[-]ata110

Yeah, "people who think Singularitarians think that" is what I meant.

I've actually met exactly one something-like-a-Singularitarian who did think something-like-that — it was at one of the Bay Area meetups, so you may or may not have talked to him, but anyway, he was saying that only people who invent or otherwise contribute to the development of Singularity technology would "deserve" to actually benefit from a positive Singularity. He wasn't exactly saying he believed that the nonbelievers would be left to languish when cometh the Singularity, but he seemed to be saying that they should.

Also, I think he tried to convert me to Objectivism.

-1timtyler
Technological progress has increased weath inequality a great deal so far. Machine intelligence probably has the potential to result in enormous weath inequality.
2WrongBot
How, in a post-AGI world, would you define wealth? Computational resources? Matter? I don't think there's any foundation for speculation on this topic at this time.
3khafra
Unless we get a hard-takeoff singleton, which is admittedly the SIAI expectation, there will be massive inequality, with a few very wealthy beings and average income barely above subsistence. Thus saith Robin Hanson, and I've never seen any significant holes poked in that thesis.
1WrongBot
Robin Hanson seems to be assuming that human preferences will, in general, remain in their current ranges. This strikes me as unlikely in the face of technological self-modification.
3khafra
I've never gotten that impression. What I've gotten is that evolutionary pressures will, in the long term, still exist--even if technological self-modification leads to a population that's 99.99% satisfied to live within strict resource consumption limits, unless they harshly punish defectors the .01% with a drive for replication or expansion will overwhelm the rest within a few millenia, until the average income is back to subsistence. This doesn't depend on human preferences, just the laws of physics and natural selection.
3WrongBot
What evolutionary pressures? Even making the incredible assumption that we will continue to use sequences of genes as a large part of our identities, what's to stop a singleton of some variety from eliminating drives for replication or expansion entirely? I feel uncomfortable speculating about a post-machine-intelligence future even to this extent; this is not a realm in which I am confident about any proposition. Consequently, I view all confident conclusions with great skepticism.
5khafra
You're still not getting the breadth and generality of Hanson's model. To use recent LW terminology, it's an anti-prediction. It doesn't matter whether agents perpetuate their strategies by DNA mixing, binary fission, cellular automata, or cave paintings. Even if all but a tiny minority of posthumans self-modify not to want growth or replication, the few that don't will soon dominate the light-cone. A singleton, like I'd mentioned, is one way to avert this. Universal extinction and harsh, immediate punishment of expansion-oriented agents are the only others I see.
1WrongBot
You (or Robin, I suppose) are just describing a many-agent prisoner's dilemma. If TDT agents beat the dilemma by cooperating with other TDT agents, then any agents that started out with a different decision theory will have long since self-modified to use TDT. Alternately, if there is no best decision theoretical solution to the prisoner's dilemma, then we probably don't need to worry about surviving to face this problem.
0khafra
Now, there's a generalized answer. It even covers the possibility of meeting aliens--finding TDT is a necessary condition for reaching the stars. Harsh punishment of inconsiderate expanders might still be required, but there could be a stable equilibrium without ever actually inflicting that punishment. That's a new perspective for me, thanks!
0WrongBot
To be even more general, suppose that there is at least one thing X that is universally necessary for effective superintelligences to function. X might be knowledge of the second law of thermodynamics, TDT, a computational substrate of some variety, or any number of other things. There are probably very many such X's, many of which are entirely non-obvious to any entity that is not itself a superintelligence (i.e. us). Furthermore, there may be at least one thing Y that is universally incompatible with effective superintelligence. Y might be an absolute belief in the existence of the deity Thor or desiring only to solve the Halting Problem using a TM-equivalent. For the Hansonian model to hold, all X's and no Y's must be compatible with the desire and ability to expand and/or replicate. This argument is generally why I dislike speculating about superintelligences. It is impossible for ordinary humans to have exhaustive (or even useful, partial) knowledge of all X and all Y. The set of all things Y in particular may not even be enumerable.
0wedrifid
We cannot be sure that there are difficulties beyond our comprehension but we are certainly able to assign probabilities to that hypothesis based on what we know. I would be justifiably shocked if something we could call a super-intelligence couldn't be formed based on knowledge that is accessible to us, even if the process of putting the seed of a super-intelligence together is beyond us. Humans aren't even remotely optimised for generalised intelligence, it's just a trick we picked up to, crudely speaking, get laid. There is no reason that a intelligence of the form "human thinking minus the parts that suck and a bit more of the parts that don't suck" couldn't be created using the knowledge available to us and that is something we can easily place a high probability on. Then you run the hardware at more than 60hz.
0WrongBot
Oh, I agree. We just don't know what self-modifications will be necessary to achieve non-speed-based optimizations. To put it another way, if superintelligences are competing with each other and self-modifying in order to do so, predictions about the qualities those superintelligences will possess are all but worthless.
0wedrifid
On this I totally agree!
0wedrifid
Your point is spot on. Competition can not be relied on to produce adaptation if someone wins the competition once and for all.
1Vladimir_Nesov
Control, owned by preferences.
0timtyler
I wasn't trying to make an especially long-term prediction: "We saw the first millionaire in 1716, the first billionaire in 1916 - and can expect the first trillionaire within the next decade - probably before 2016."
7WrongBot
1. Inflation. 2. The richest person on earth currently has a net worth of $53.5 billion. 3. The greatest peak net worth in recorded history, adjusted for inflation, was Bill Gates' $101 billion, which was ten years ago. No one since then has come close. A 10-fold increase in <6 years strikes me as unlikely. 4. In any case, your extrapolated curve points to 2116, not 2016. I am increasingly convinced that your comments on this topic are made in less than good faith.
0timtyler
Yes, the last figure looks wrong to me too - hopefully I will revisit the issue. Update 2011-05-30: yes: 2016 was a simple math mistake! I have updated the text I was quoting from to read "later this century". Anyway, the huge modern wealth inequalities are well established - and projecting them into the future doesn't seem especially controversial. Today's winners in IT are hugely rich - and tomorrow's winners may well be even richer. People thinking something like they will "be on the inside track when the singularity happens" would not be very surprising.
6WrongBot
Projecting anything into a future with non-human intelligences is controversial. You have made an incredibly large assumption without realizing it. Please update.
0timtyler
If you actually want your questions answered, then money is society's representation of utility - and I think there will probably be something like that in the future - no matter how far out you go. What you may not find further out is "people". However, I wasn't talking about any of that, really. I just meant while there are still money and people with bank accounts around.
0WrongBot
A few levels up, you said, My dispute is with the notion that people with bank accounts and machine intelligence will coexist for a non-trivial amount of time.
0timtyler
We have been building intelligent machines for many decades now. If you are talking about something that doesn't yet exist, I think you would be well advised to find another term for it.
0WrongBot
Apologies; I assumed you were using "machine intelligence" as a synonym for AI, as wikipedia does.
1timtyler
Machine intelligence *is - more-or-less - a synonym for artificial intelligence. Neither term carries the implication of human-level intelligence.
0[anonymous]
We don't really have a good canonical term for "AI or upload".
-3timtyler
What about the recent "forbidden topic"? Surely that is a prime example of this kind of thing.
-8[anonymous]
0timtyler
The outside view of the pitch: * DOOM! - and SOON! * GIVE US ALL YOUR MONEY; * We'll SAVE THE WORLD; you'll LIVE FOREVER in HEAVEN; * Do otherwise and YOU and YOUR LOVED ONES will suffer ETERNAL OBLIVION! Maybe there are some bits missing - but they don't appear to be critical components of the pattern. Indeed, this time there are some extra features not invented by those who went before - e.g.: * We can even send you to HEAVEN if you DIE a sinner - IF you PAY MORE MONEY to our partner organisation.

Do otherwise and YOU and YOUR LOVED ONES will suffer ETERNAL OBLIVION.

This one isn't right, and is a big difference between religion and threats like extinction-level asteroids or AI disasters: one can free-ride if that's one's practice in collective action problems.

Also: Rapture of the Nerds, Not

-7timtyler
-11timtyler
2cousin_it
I don't understand why downvote this. It does sound like an accurate representation of the outside view.

This whole "outside view" methodology, where you insist on arguing from ignorance even where you have additional knowledge, is insane (outside of avoiding the specific biases such as planning fallacy induced by making additional detail available to your mind, where you indirectly benefit from basing your decision on ignorance).

In many cases outside view, and in particular reference class tennis, is a form of filtering the evidence, and thus "not technically" lying, a tool of anti-epistemology and dark arts, fit for deceiving yourself and others.

-12timtyler

We all already know about this pattern match. Its reiteration is boring and detracts from the conversation.

1timtyler
If this particular critique has been made more clearly elsewhere, perhaps let me know, and I will happily link to there in the future. Update 2011-05-30: There's now this recent article: The “Rapture” and the “Singularity” Have Much in Common - which makes a rather similar point.
6Unknowns
It may have been downvoted for the caps.
3[anonymous]
Given that a certain fraction of comments are foolish, you can expect that an even larger fraction of votes are foolish, because there are fewer controls on votes (e.g. a voter doesn't risk his reputation while a commenter does).
3RHollerith
Which is why Slashdot (which was a lot more worthwhile in the past than it is now) introduced voting on how other people vote (which Slashdot called metamoderation). Worked pretty well: the decline of Slashdot was mild and gradual compared to the decline of almost every other social site that ever reached Slashdot's level of quality.
-1timtyler
Yes: votes should probably not be anonymous - and on "various other" social networking sites, they are not.
0RHollerith
Metafilter, for one. It is hard for an online community to avoid becoming worthless, but Metafilter has avoided that for 10 years.
3Perplexed
Perhaps downvoted for suggesting that the salvation-for-cash meme is a modern one. I upvoted, though.
0timtyler
Hmm - I didn't think of that. Maybe deathbed repentance is similar as well - in that it offers sinners a shot at eternal bliss in return for public endorsement - and maybe a slice of the will.
0[anonymous]
We all already know about this pattern match. Reiterating it is boring and detracts from the conversation, and I downvote any such comment I see.
-2TheAncientGeek
Pattern completion isnt always wrong.
6Strange7
What about less-smart people? I mean, self-motivated idealistic genius nerds are certainly necessary for the core functions of programming an FAI, but any sufficiently large organization also needs a certain number of people who mostly just file paperwork, follow orders, answer the phone, etc. and things tend to work out more efficiently when those people are primarily motivated by the organization's actual goals rather than it's willingness to pay.
2HughRistik
Good point. It's the people in the <130 range that SIAI needs to figure out how to attract. That's where you find people like journalists and politicians.
7wedrifid
You also find a lot of journalists and politicians in the 130 to 160 range but the important thing with those groups is that they optimise their beliefs and expressions thereof for appeal to a < 130 range audience.
4multifoliaterose
Leaving aside the question of whether such apparently strong estimation is warranted in the case at hand; I would suggest that there's a serious possibility that the social taboo that you allude to is adaptive; that having a very high opinion of oneself (even if justified) is (on account of the affect heuristic) conducive to seeing a halo around oneself, developing overconfidence bias, rejecting criticisms prematurely, etc. leading to undesirable epistemological skewing. Same here. It's easy to blunt this signal. Suppose that any of: 1. A billionaire decided to devote most of his or her wealth to funding Friendly AI research. 2. A dozen brilliant academics became interested in and started doing Friendly AI research. 3. The probability of Friendly AI research leading to a Friendly AI is sufficiently low so that another existential risk reduction effort (e.g. pursuit of stable whole brain emulation) is many orders of magnitude more cost-effective at reducing existential risk than Friendly AI research. Then the Eliezer would not (by most estimations) be the highest utilitarian expected value human in the world. If he were to mention such possibilities explicitly this would greatly mute the undesired connotations.
8Eliezer Yudkowsky
If I thought whole-brain emulation were far more effective I would be pushing whole-brain emulation, FOR THE LOVE OF SQUIRRELS!
3multifoliaterose
Good to hear from you :-) 1. My understanding is that at present there's a great deal of uncertainty concerning how future advanced technologies are going to develop (I've gotten an impression that e.g. Nick Bostrom and Josh Tenenbaum hold this view). In view of such uncertainty, it's easy to imagine new data emerging over the next decades that makes it clear that pursuit of whole-brain emulation (or some currently unimagined strategy) is a far more effective strategy for existential risk reduction than Friendly AI research. 2. At present it looks to me like a positive singularity is substantially more likely to occur starting with whole-brain emulation than with Friendly AI. 3. Various people have suggested to me that initially pursuing Friendly AI might have higher expected value on the chance that it turns out to be easy. So I could imagine that it's rational for you personally to focus your efforts on Friendly AI research (EDIT: even if I'm correct in my estimation in the above point). My remarks in the grandparent above were not intended as a criticism of your strategy. 4. I would be interested in hearing more about your own thinking about the relative feasibility of Friendly AI vs. stable whole-brain emulation and current arbitrage opportunities for existential risk reduction, whether on or off the record.
4ata
That's an interesting claim, and you should post your analysis of it (e.g. the evidence and reasoning that you use to form the estimate that a positive singularity is "substantially more likely" given WBE).
1multifoliaterose
There's a thread with some relevant points (both for and against) titled Hedging our Bets: The Case for Pursuing Whole Brain Emulation to Safeguard Humanity's Future. I hadn't looked at the comments until just now and still have to read them all; but see in particular a comment by Carl Shulman. After reading all of the comments I'll think about whether I have something to add beyond them and get back to you.
4CarlShulman
You may want to read this paper I presented at FHI. Note that there's a big difference between the probability of risk conditional on WBE coming first or AI coming first and marginal impact of effort. In particular some of our uncertainty is about logical facts about the space of algorithms and technology landscape, and some of it is about the extent and effectiveness of activism/intervention.
3multifoliaterose
Thanks for the very interesting reference! Is it linked on the SIAI research papers page? I didn't see it there. I appreciate this point which you've made to me previously (and which appears in your comment that I linked above!).
2Vladimir_Nesov
Do you mean that the role of ems is in developing FAI faster (as opposed to biological-human-built FAI), or are you thinking of something else? If ems merely speed time up, they don't change the shape of FAI challenge much, unless (and to the extent that) we leverage them in a way we can't for the human society to reduce existential risk before FAI is complete (but this can turn out worse as well, ems can well launch the first arbitrary-goal AGI).
6ata
That's the main thing that's worried me about the possibility of ems coming first. But it depends on who is able to upload and who wants to, I suppose. If an average FAI researcher is more likely to upload, increase their speed, and possibly make copies of themselves than an average non-FAI AGI researcher, then it seems like that would be a reduction in risk. I'm not sure whether that would be the case — a person working on FAI is likely to consider their work to be a matter of life and death, and would want all the speed increases they could get, but an AGI researcher may feel the same way about the threat to their career and status posed by the possibility of someone else getting to AGI first. And if uploading is very expensive at first, it'll only be the most well-funded AGI researchers (i.e. not SIAI and friends) who will have access to it early on and will be likely to attempt it (if it provides enough of a speed increase that they'd consider it to be worth it). (I originally thought that uploading would be of little to no help in increasing one's own intelligence (in ways aside from thinking the same way but faster), since an emulation of a brain isn't automatically any more comprehensible than an actual brain, but now I can see a few ways it could help — the equivalent of any kind of brain surgery could be attempted quickly, freely, and reversibly, and the same could be said for experimenting with nootropic-type effects within the emulation. So it's possible that uploaded people would get somewhat smarter and not just faster. Of course, that's only soft self-improvement, nowhere near the ability to systematically change one's cognition at the algorithmic level, so I'm not worried about an upload bootstrapping itself to superintelligence (as some people apparently are). Which is good, since humans are not Friendly.)
4multifoliaterose
There's a lot to respond to here. Some quick points: 1. It should be born in mind that greatly increased speed and memory may by themselves strongly affect a thinking entity. I imagine that if I could think a million times as fast I would think a lot more carefully about my interactions with the outside world than I do now. 2. I don't see any reason to think that SIAI will continue to be the only group thinking about safety considerations. If nothing else, SIAI or FHI can raise awareness of the dangers of AI within the community of AI researchers. 3. Assuming that brain uploads precede superhuman artificial intelligence, it would obviously be very desirable to have the right sort of human uploaded first. 4. I presently have a very dim view as to the prospects for modern day humans developing Friendly AI. This skepticism is the main reason why I think that pursuing whole-brain emulations first is more promising. See the comment by Carl that I mentioned in response to Vladimir Nesov's question. Of course, my attitude on this point is subject to change with incoming evidence.
3CarlShulman
Sped-up ems have slower computers relative to their thinking speed. If Moore's Law of Mad Science means that increasing computing power allows researchers to build AI with less understanding (and thus more risk of UFAI), then a speedup of researchers relative to computing speed makes it more likely that the first non-WBE AIs will be the result of a theory-intensive approach with high understanding. Anders Sandberg of FHI and I are working on a paper exploring some of these issues.
3Vladimir_Nesov
This argument lowers the estimate of danger, but AIs developed on relatively slow computers are not necessarily theory-intense, could also be coding-intense, which leads to UFAI. And theory-intense doesn't necessarily imply adequate concern about AI's preference.
2multifoliaterose
My idea here is the same as the one that Carl Shulman mentioned in a response to one of your comments from nine months ago.
-7timtyler
-1halcyon
In cases like this, I find ethics grounded in utilitarianism to be despicably manipulative positions. You are not treating people as rational agents, but pandering to their lack of virtue so as to recruit them as pawns in your game. If that's how you're going to play, why not manufacture evidence in support of your position if you're Really Sure your assessment is accurate? A clear line of division between "pandering: acceptable" & "evidence manufacture: unacceptable" is nothing but a temporary, culturally contingent consensus caring nothing for reason or consistency. To predict the future, see the direction in which the trend is headed. No, I would scrupulously adhere to a position of utmost sincerity. Screw the easily offended customers. If this causes my downfall, so be it. That outcome is acceptable because personally, if my failure is caused by honesty and goodwill rather than incompetence, I would question if such a world is worth saving to begin with. I mean, if that is what this enlightened society is like and wants to be like, then I can rather easily imagine our species eventually ending up as the aggressors in one of those alien invasion movies like Independence Day. I keep wondering why, if they evolved in a symbiotic ecosystem analogous to ours, one morally committed individual among their number didn't wipe out their own race and rid the galaxy of this aimless, proliferating evil. It'd be better still to let them be smothered peacefully under their own absence of self-reflection and practice of rewarding corruption, without going out of your way to help them artificially reach a position of preeminence from which to bully others.

A number of people have mentioned the seemingly-unimpeachable reputation of the Future of Humanity Institute without mentioning that its director, Nick Bostrom, fairly obviously has a high opinion of Eliezer (e.g., he invited him to contribute not one but two chapters to the volume on Global Catastrophic Risks). Heuristically, if I have a high opinion of Bostrom and the FHI project, that raises my opinion of Eliezer and decreases the probability of Eliezer-as-crackpot.

Well, in the category of "criticisms of SIAI and/or Eliezer", this text is certainly among the better ones. I could see this included on a "required reading list" of new SIAI employees or something.

But since we're talking about a Very Important Issue, i.e. existential risks, the text might have benefited from some closing warnings, that whatever people's perceptions of SIAI, it's Very Important that they don't neglect being very seriously interested in existential risks because of issues that they might perceive a particular organization working on the topic to have (and that it might also actually have, but that's not my focus in this comment).

I.e. if people think SIAI sucks and shouldn't be supported, they should anyway be very interested in supporting the Future of Humanity Institute at Oxford, for example. Otherwise they're demonstrating very high levels of irrationality, and with regard to SIAI, are probably just looking for plausible-sounding excuses to latch onto for why they shouldn't pitch in.

Not to say that the criticism you presented mightn't be very valid (or not; I'm not really commenting on that here), but it would be very important for people to f... (read more)

-1timtyler
Very important to you - maybe. You aware, I presume, that for most people, the end of the world is not high on their agenda. It is evidently not "very important" to them - or they would spend more time on it. Basic biology explains this phenomenon, as I have previously explained: "Organisms can be expected to concentrate on producing offspring - not indulging paranoid fantasies about their whole species being wiped out!"
2Aleksei_Riikonen
Are you aware that most species that have ever lived have indeed been wiped out? Not thinking about such possibilities worked well for them, eh? EDIT: And of course we can also present scholarly analyses of why extinction in the case of our species is not particularly unlikely: http://www.nickbostrom.com/fut/evolution.html
-3timtyler
If you mean to imply that thinking about such possibilities would have helped them all to survive, then that doesn't seem right. If new species keep being born (as happens naturally), other ones seem practically bound to die out - due to resource competiton in a limited ecosystem. Hypothetical contemplation of their own species-level mortality seems unlikely to have helped - and might well have hindered their survival chances.
1Aleksei_Riikonen
Thinking about such things is the necessary first step to preventing such new species from arising that would make you extinct. So yes, if they had thought about these things competently enough, and otherwise been competent enough, it would have enabled them to survive. Doesn't seem very smart of you to argue against thinking. If you don't think, you're certainly even more screwed than with thinking.
0timtyler
The "most species that have ever lived" that you mentioned were not capable of preventing new species from arising - because that happens naturally all the time. If you introduce this hypothetical, it seems as though you have to abandon your original argument. It is thinking too much about events that you have little control over that can be bad. Also, in biology, more thinking than normal is not good, on average - thoughts are costly and there is an economic tradeoff.
[-][anonymous]180

I don't think there's any point doing armchair diagnoses and accusing people of delusions of grandeur. I wouldn't go so far as to claim that Eliezer needs more self-doubt, in a psychological sense. That's an awfully personal statement to make publicly. It's not self-confidence I'm worried about, it's insularity.

Here's the thing. The whole SIAI project is not publicly affiliated with (as far as I've heard) other, more mainstream institutions with relevant expertise. Universities, government agencies, corporations. We don't have guest posts from Dr. X or Think Tank Fellow Y. The ideas related to friendly AI and existential risk have not been shopped to academia or evaluated by scientists in the usual way. So they're not being tested stringently enough.

It's speculative. It feels fuzzy to me -- I'm not an expert in AI, but I have some education in math, and things feel fuzzy around here.

If you want to claim you're working on a project that may save the world, fine. But there's got to be more to show for it, sooner or later, than speculative essays. At the very least, people worried about unfriendly AI will have to gather data and come up with some kind of statistical stu... (read more)

The whole SIAI project is not publicly affiliated with (as far as I've heard) other, more mainstream institutions with relevant expertise. Universities, government agencies, corporations. We don't have guest posts from Dr. X or Think Tank Fellow Y.

According to the about page, LW is brought to you by the Future of Humanity Institute at Oxford University. Does this count? Many Dr. Xes have spoken at the Singularity Summits.

At the very least, people worried about unfriendly AI will have to gather data and come up with some kind of statistical study that gives evidence of a threat!

It's not clear how one would use past data to give evidence for or against a UFAI threat in any straightforward way. There's various kinds of indirect evidence that could be presented, and SIAI has indeed been trying more in the last year or two to publish articles and give conference talks presenting such evidence.

Points that SIAI would do better if it had better PR, had more transparency, published more in the scientific literature, etc., are all well-taken, but these things use limited resources, which to me makes it sound strange to use them as arguments to direct funding elsewhere.

6[anonymous]
My post was by way of explaining why some people (including myself) doubt the claims of SIAI. People doubt claims when, compared to other claims, they're not justified as rigorously, or haven't met certain public standards. Why do I agree with the main post that Eliezer isn't justified in his opinion of his own importance (and SIAI's importance)? Because there isn't (yet) a lot beyond speculation here. I understand about limited resources. If I were trying to run a foundation like SIAI, I might do exactly what it's doing, at first, and then try to get the academic credentials. But as an outside person, trying to determine: is this worth my time? Is this worth further study? Is this a field I could work in? Is this worth my giving away part of my (currently puny) income in donations? I'm likely to hold off until I see something stronger. And I'm likely to be turned off by statements with a tone that assumes anyone sufficiently rational should already be on board. Well, no! It's not an obvious, open-and shut deal. What if there were an organization comprised of idealistic, speculative types, who, unknowingly, got themselves to believe something completely false based on sketchy philosophical arguments? They might look a lot like SIAI. Could an outside observer distinguish fruitful non-mainstream speculation from pointless non-mainstream speculation?
1timtyler
I think they are working on their "academic credentials": http://singinst.org/grants/challenge ...lists some 13 academic papers under various stages of development.
1torekp
Thanks for that last link. The paper on Changing the frame of AI futurism is extremely relevant to this series of posts.
0[anonymous]
I contacted Nick Bostrom about this and he said that there’s no formal relationship between FHI and SIAI. See my comments here, here and here.

Here's the thing. The whole SIAI project is not publicly affiliated with (as far as I've heard) other, more mainstream institutions with relevant expertise.

LessWrong is itself a joint project of the SIAI and the Future of Humanity Institute at Oxford. Researchers at the SIAI have published these academic papers. The Singularity Summit's website includes a lengthy list of partners, including Google and Scientific American.

The SIAI and Eliezer may not have done the best possible job of engaging with the academic mainstream, but they haven't done a terrible one either, and accusations that they aren't trying are, so far as I am able to determine, factually inaccurate.

6Perplexed
But those don't really qualify as "published academic papers" in the sense that those terms are usually understood in academia. They are instead "research reports" or "technical reports". The one additional hoop that these high-quality articles should pass through before they earn the status of true academic publications is to actually be published - i.e. accepted by a reputable (paper or online) journal. This hoop exists for a variety of reasons, including the claim that the research has been subjected to at least a modicum of unbiased review, a locus for post-publication critique (at least a journal letters-to-editor column), and a promise of stable curatorship. Plus inclusion in citation indexes and the like. Perhaps the FHI should sponsor a journal, to serve as a venue and repository for research articles like these.
1CarlShulman
There are already relevant niche philosophy journals (Ethics and Information Technology, Minds and Machines, and Philosophy and Technology). Robin Hanson's "Economic Growth Given Machine Intelligence" has been accepted in an AI journal, and there are forecasting journals like Technological Forecasting and Social Change. For more unusual topics, there's the Journal of Evolution and Technology. SIAI folk are working to submit the current crop of papers for publication.
1Perplexed
Cool!
4[anonymous]
Okay, I take that back. I did know about the connection between SIAI and FHI and Oxford. What are these academic papers published in? A lot of them don't provide that information; one is in Global Catastrophic Risks. At any rate, I exaggerated in saying there isn't any engagement with the academic mainstream. But it looks like it's not very much. And I recall a post of Eliezer's that said, roughly, "It's not that academia has rejected my ideas, it's that I haven't done the work of trying to get academia's attention." Well, why not?
5WrongBot
Limited time and more important objectives, I would assume. Most academic work is not substantially better than trial-and-error in terms of usefulness and accuracy; it gets by on volume. Volume is a detriment in Friendliness research, because errors can have large detrimental effects relative to the size of the error. (Like the accidental creation of a paperclipper.)
1Eliezer Yudkowsky
If you want it done, feel free to do it yourself. :)
0wedrifid
... particularly in as much as they have become (somewhat) obsolete.
0MatthewBaker
Can you clarify please?
1wedrifid
Basically, no. Whatever I meant seems to have been lost to me in the temporal context.
0MatthewBaker
No worries, I do the same thing sometimes.
6Morendil
Possibly because this blog is Less Wrong, positioned as "a community blog devoted to refining the art of human rationality", and not as the SIAI blog, or an existential risk blog, or an FAI blog.
4multifoliaterose
I respectfully disagree with this statement, at least as an absolute. I believe that: (A) In situations in which people are making significant life choices based on person X's claims and person X exhibits behavior which is highly correlated with delusions of grandeur, it's appropriate to raise the possibility that person X's claims arise from delusions of grandeur and ask that person X publicly address this possibility. (B) When one raises the possibility that somebody is suffering from delusions of grandeur, this should be done in as polite and nonconfrontational way as possible given the nature of the topic. I believe that if more people adopted these practices, this would would raise the sanity waterline. I believe that the situation with respect to Eliezer and portions of the LW community is as in (A) and that I made a good faith effort at (B).
3wedrifid
I agree with your conclusion but not this part: I categorically do not want statistical studies of the type you mention done. I do want solid academic research done but not experiments. Some statistics on, for example, human predictions vs actual time till successful completion on tasks of various difficulties would be useful. But these do not appear to be the type of studies you are asking for, and nor do they target the most significant parts of the conclusion. You are not entitled to that particular proof. EDIT: The 'entitlement' link was broken.
3timtyler
There's these fellows: * http://singinst.org/aboutus/advisors Some of them have contributed here: * http://singinst.org/media/interviews
1[anonymous]
I agree with your conclusion but not this part: I categorically do not want statistical studies of the type you mention done. I do want solid academic research done but not experiments. Some statistics on, for example, human predictions vs actual time till successful completion on tasks of various difficulties would be useful. But these do not appear to be the type of studies you are asking for, and nor do they target the most significant parts of the conclusion. [You are not entitled to that particular proof]http://lesswrong.com/lw/1ph/youre_entitled_to_arguments_but_not_that/).
0Perplexed
I only wish it were possible to upvote this comment more than once.

I feel that perhaps you haven't considered the best way to maximise your chance of developing Friendly AI if you were Eliezer Yudkowsky; your perspective is very much focussed on how you see it lookin in from the outside. Consider for a moment that you are in a situation where you think you can make a huge positive impact upon the world, and have founded an organisation to help you act upon that.

Your first, and biggest problem is getting paid. You could take time off to work on attaining a fortune through some other means but this is not a certain bet, and will waste years that you could be spending working on the problem instead. Your best bet is to find already wealthy people who can be convinced that you can change the world, that it's for the best, and that they should donate significant sums of money to you, unless you believe this is even less certain than making a fortune yourself. There's already a lot of people in the world with the requisite amount of money to spare. I think seeking donations is the more rational path.

Now, given that you need to persuade people of the importance of your brilliant new idea which no one has really been considering before, and that to most ... (read more)

0TheAncientGeek
A display of confidence is a good way of getting people on your side if you are right,. It is also a good way of ovwrestimating whether you are right or not.

How would you address this?

http://scienceblogs.com/pharyngula/2010/08/kurzweil_still_doesnt_understa.php

It seems to me like PZ Meyers really doesn't understand information theory. He's attacking Kurzweil and calling him a kook. Initially due to a relatively straightforward complexity estimate.

And I'm pretty confident that Myers is wrong on this, unless there is another information rich source of inheritance besides DNA, which Meyers knows about but Kurzweil and I do not.

This looks to me like a popular science blogger doing huge PR damage to everything singularity related, and being wrong about it. Even if he is later convinced of this point.

I don't see how to avoid this short of just holding back all claims which seem exceptional and that some 'reasonable' person might fail to understand and see as a sign of cultishness. If we can't make claims as basic as the design of the brain being in the genome, then we may as well just remain silent.

But then we wouldn't find out if we're wrong, and we're rationalists.

For instance, you can't measure the number of transistors in an Intel CPU and then announce, "A-ha! We now understand what a small amount of information is actually required to create all those operating systems and computer games and Microsoft Word, and it is much, much smaller than everyone is assuming."

This analogy made me cringe. Myers is disagreeing with the claim that human DNA completely encodes the structure and functioning of the human brain: the hardware and software, roughly. Looking at the complexity of the hardware and making claims about the complexity of the software, as he does here, is completely irrelevant to his disagreement. It serves only to obscure the actual point under debate, and demonstrates that he has no idea what he's talking about.

7Risto_Saarelma
There seems to be a culture clash between computer scientists and biologists with this matter. DNA bit length as a back-of-the-envelope complexity estimate for a heavily compressed AGI source seems obvious to me, and, it seems, to Larry Page. Biologists are quick to jump to the particulars of protein synthesis and ignore the question of extra information, because biologists don't really deal with information theoretical existence proofs. It really doesn't help the matter that Kurzweil threw out his estimate when talking about getting at AGI by specifically emulating the human brain, instead of just trying to develop a general human-equivalent AI using code suitable for the computation platform used. This seems to steer most people into thinking that Kurzweil was thinking of using the DNA as literal source code instead of just a complexity yardstick. Myers seems to have pretty much gone into his creationist-bashing attack mode on this, so I don't have a very high hopes for any meaningful dialogue from him.
4whpearson
I'm still not sure what people are trying to say with this. Because the kolmogorov complexity of the human brain given the language of the genetic code and physics is low, therefore X? What is that X precisely? Because of kolmogorov complexities additive constant, which could be anything from 0 to 3^^^3 or higher, I think it only gives us weak evidence for the amount of code we should expect it to take to code an AI on a computer. It is even weaker evidence for the amount of code it would take to code for it with limited resources. E.g. the laws of physics are simple and little information is taken from the womb, but to create an intelligence from them might require a quantum computer the size of the human head to decompress the compressed code. There might be short cuts to do it, but they might be of vastly greater complexity. We tend to ignore additive constants when talking about Complexity classes, because human designed algorithms tend not to have huge additive constants. Although I have come across some in my time such as this...
4Emile
We have something like this going on like: discrete DNA code -> lots of messy chemistry and biology -> human intelligence and we're comparing it to : discrete computer code -> computer -> human intelligence Kurzweil is arguing that the size of the DNA code can tell us about the max size of the computer code needed to run an intelligent brain simulation (or a human-level AI), and PZ Myers is basically saying "no, 'cause that chemistry and biology is really really messy". Now, I agree that the computer code and the DNA code are very very different ("a huge amount of enzymes interacting with each other in 3D real time" isn't the kind of thing you easily simulate on a computer), and the additive constant for converting one into the other is likely to be pretty darn big. But I also don't see a reason for intelligence to be easier to express with messy biology and chemistry than with computer code. The things about intelligence that are the closest to biology (interfacing with the real world, how one neuron functions) are also the kind of things that we can already do quite well with computer programs. There are some things that are "natural" to code in Prolog, but not natural in Fortran, fotran. So a short program in prolog might require a long program in Fotran to do the same thing, and for different programs it might be the other way around. I don't see any reason to think that it's easier to encode intelligence in DNA than it is in computer code. (Now, Kurzweil may be overstating his case when he talks about "compressed" DNA, because to be fair you should compare that to compressed (or compiled) computer code, which translates to much more actual code. I still think the size of the DNA is a very reasonable upper limit, especially when you consider that the DNA was coded by a bloody idiot whose main design pattern is "copy-and-paste", resulting in the bloated code we know)
2whpearson
Do you have any reason to expect it to be the same? Do we have any reason at all? I'm not arguing that it will take more than 50MBs of code, I'm arguing that the DNA value is not informative. We are far less good at the doing the equivalent of changing neural structure or adding new neurons (we don't know why or how neurogenesis works for one) in computer programs.
3Emile
If I know a certain concept X requires 12 seconds of speech to express in English, and I don't know anything about Swahili beyond the fact that it's a human language, my first guess will be that concept X requires 12 seconds of speech to express in Swahili. I would also express compressed versions of translations in various languages of the same book to be roughly the same size. So, even with very little information, a first estimate (with a big error margin) would be that it takes as many bits to "encode" intelligence in DNA than it does in computer code. In addition, the fact that some intelligence-related abilities such as multiplying large numbers are easy to express in computer code, but rare in nature would make me revise that estimate towards "code as more expressive than DNA for some intelligence-related stuff". In addition, knowledge about the history of evolution would make me suspect that large chunks of the human genome are not required for intelligence, either because they aren't expressed, or because they only concern traits that have no impact on our intelligence beyond the fact of keeping us alive. That would also make me revise my estimate downwards for the code size needed for intelligence. None of those are very strong reasons, but they are reasons nonetheless!
0whpearson
You'd be very wrong for a lot of technical language, unless they just imported the English words whole sale. For example, "Algorithmic Information Theory," expresses a concept well but I'm guessing it would be hard to explain in Swahili. Even given that, you can expect the languages of humans to all have roughly the same length because they are generated by the roughly the same hardware and have roughly the same concerns. E.g. things to do with humans. To give a more realistic translation problem, how long would you expect it to take to express/explain any random English in C code or vice versa?
3Peter_de_Blanc
Selecting a random English sentence will introduce a bias towards concepts that are easy to express in English.
5Kingreaper
The environment is information-rich, especially the social environment. Meyers make it quite clear that interactions with the environment are an expected input of information in his understanding. Do you disagree with information input from the environment?
4JamesAndrix
Yes, I disagree. If he's not talking about some stable information that is present in all environments that yield intelligent humans, then what's important is a kind of information that can be mass generated at low complexity cost. Even language exposure is relatively low complexity, and the key parts might be inferable from brain processes. And we already know how to offer a socially rich environment, so I don't think it should add to the complexity costs of this problem. And I think a reverse engineering of a newborn baby brain would be quite sufficient for kurzweil's goal. In short: we know intelligent brains get reliably generated. We know it's very complex. The source of that complexity must be something information rich, stable, and universal. I know of exactly one such source. Right now I'm reading myers argument as "a big part of human heredity is memetic rather than just genetic, and there is complex interplay between genes and memes, so you've got to count the memes as part of the total complexity." I say that Kurzweil is trying to create something compatible with human memes in the first plalce, so we can load them the same way we load children (at worst) And even some classes of memes (age appropriate language exposure) do interact tightly with genes, their information content is not all that high.
-1whpearson
While doable this seems like a very time consuming project and potentially morally dubious. How do you know when you have succeeded and not got a mildly brain damaged one, because you have missed an important detail for language learning? We really don't want to be running multi year experiments, where humans have to interact with infant machines, that would be ruinously expensive. The quicker you can evaluate the capabilities of the machine the better.
-1JamesAndrix
Well in Kurzweils' case, you'd look at the source code and debug it to make sure it's doinjg everything it's supposed to, because he's no dealing with a meat brain. I guess my real point is that language learning should not be tacked on to the problem of reverse engineering the brain, If he makes something that is as capable of learning, that's a win for him. (Hopefully he also reverse engineers all of human morality.)
0whpearson
You are assuming the program found via the reverse engineering process is human understandable.... what if it is a strange cellular automata with odd rules. Or an algorithm with parameters you don't know why they are what they are. Language is an important part of learning for humans. Imagine trying to learn chess if no one explained the legal moves. Something without the capability for language isn't such a big win IMHO.
0JamesAndrix
I think we might have different visions of what this reverse engineering would entail, By my concept, if you don't understand the function of the program you wrote, you're not done reverse engineering. I do think that something capable of learning language would be necessary for a win. but the information content of the language does not count towards the complexity estimate of the thing capable of learning langauge.
5Emile
I see it that way too. The DNA can give us an upper bound on the information needed to create a human brain, but PZ Myers reads that as "Kurzweil is saying we will be able to take a strand of DNA and build a brain from that in the next 10 years!", and then procede to attack that straw man. This, however: ... I am quite enclined to trust. I would trust it more if it wasn't followed by wrong statements about information theory (that seem wrong to me, at least). Looking at the comments is depressing. I wish there was some "sane" ways for two communities (readers of PZ Myers and "singularitarians") to engage without it degenerating into name-calling. Though there are software solutions for that (takeonit and other stuff that's been discussed here), it wouldn't help either if the "leaders" (PZ Myers, Kurzweil, etc.) were a bit more responsible and made a genuine effort to acknowledge the other's points when there are strong. So they could converge or at least agree to disagree on something narrow. But nooo, it's much more fun to get angry, and it gets you more traffic too!
0RobinZ
Why do you say this? If humans were designed by human engineers, the 'blueprints' would actually be complete blueprints, sufficient unto the task of determining the final organism ... but they weren't. There's no particular reason to doubt that a significant amount of the final data is encoded in the gestational environment.
5Emile
I'm not sure about what you mean about the "complete blueprints" - I agree that the DNA isn't a complete blueprint, and that an alien civilization with a different chemistry would (probably) find it impossible to rebuild a human if they were just given it's DNA. The gestational environment is essential, I just don't think it encodes much data on the actual working of the brain. It seems to me that the interaction between the baby and the gestational environment is relatively simple, at least compared to organ development and differentiation. There are a lot of essential things for it to go right, and hormones and nutrients, but 1) I don't see a lot of information transfer in there ("making the brain work a certain way" as opposed to "making the brain work period"), and 2) A lot of the information on how that works is probably encoded in the DNA too. I would say that the important bits that may not be in the DNA (or in mitocondrial DNA) are the DNA interpretation system (transcription, translation).
1RobinZ
That's a strong point, but I think it's still worth bearing in mind that this subject is P. Z. Myers' actual research focus: developmental biology. It appears to me that Kurzweil should be getting Myers' help revising his 50 MB estimate*, not dismissing Myers arguments as misinformed. Yes, Myers made a mistake in responding to a summary secondhand account rather than Kurzweil's actual position, but Kurzweil is making a mistake if he's ignoring expert opinion on a subject directly relating to his thesis. * By the way: 50 MB? That's smaller than the latest version of gcc! If that's your complexity estimate, the complexity of the brain could be dominated by the complexity of the gestational environment!
1Emile
I agree that Kurzweil could have acknowledged P.Z.Myers' expertise a bit more, especially the "nobody in my field expects a brain simulation in the next ten years" bit. 50 MB - that's still a hefty amount of code, especially if it's 50MB of compiled code and not 50 MB of source code (comparing the size of the source code to the size of the compressed DNA looks fishy to me, but I'm not sure Kurzweil has been actually doing that - he's just been saying "it doesn't require trillions of lines of code"). Is the size of gcc the source code or the compiled version? I didn't see that info on Wikipedia, and don't have gcc on this machine.
3timtyler
As I see it, Myers delivered a totally misguided rant. When his mistakes were exposed he failed to apologise. Obviously, there is no such thing as bad publicity.
1RobinZ
I'm looking at gcc-4.5.0.tar.gz.
4Emile
That includes the source code, the binaries, the documentation, the unit tests, changelogs ... I'm not surpised it's pretty big! I consider it pretty likely that it's possible to program a human-like intelligence with a compressed source code of less than 50 MB. However, I'm much less confident that the source code of the first actual human-like intelligence coded by humans (if there is one) will be that size.
5Perplexed
To the contrary, there is every reason to doubt that. We already know that important pieces of the gestational environment (the genetic code itself, core metabolism, etc.) are encoded in the genome. By contrast, the amount of epigenetic information that we know of is miniscule. It is, of course, likely that we will discover more, but it is very unlikely that we will discover much more. The reason for this skepticism is that we don't know of any reliable epigenetic means of transmitting generic information from generation to generation. And the epigenetic information inheritance mechanisms that we do understand all require hundreds of times as much genetic information to specify the machinery as compared to the amount of epigenetic information that the machinery can transmit. To my mind, it is very clear that (on this narrow point) Kurzweil was right and PZ wrong: The Shannon information content of the genome places a tight upper bound on the algorithmic (i.e. Kolmogorov) information content of the embryonic brain. Admittedly, when we do finally construct an AI, it may take it 25 years to get through graduate school, and it may have to read thru several hundred Wikipedia equivalents to get there, but I am very confident that specifying the process for generating the structure and interconnect of the embryonic AI brain will take well under 7 billion bits.
1timtyler
I think you may have missed my devastating analysis of this issue a couple of years back: "So, who is right? Does the brain's design fit into the genome? - or not? The detailed form of proteins arises from a combination of the nucleotide sequence that specifies them, the cytoplasmic environment in which gene expression takes place, and the laws of physics. We can safely ignore the contribution of cytoplasmic inheritance - however, the contribution of the laws of physics is harder to discount. At first sight, it may seem simply absurd to argue that the laws of physics contain design information relating to the construction of the human brain. However there is a well-established mechanism by which physical law may do just that - an idea known as the anthropic principle. This argues that the universe we observe must necessarily permit the emergence of intelligent agents. If that involves a coding the design of the brains of intelligent agents into the laws of physics then: so be it. There are plenty of apparently-arbitrary constants in physics where such information could conceivably be encoded: the fine structure constant, the cosmological constant, Planck's constant - and so on. At the moment, it is not even possible to bound the quantity of brain-design information so encoded. When we get machine intelligence, we will have an independent estimate of the complexity of the design required to produce an intelligent agent. Alternatively, when we know what the laws of physics are, we may be able to bound the quantity of information encoded by them. However, today neither option is available to us." * http://alife.co.uk/essays/how_long_before_superintelligence/
3Perplexed
You suggest that the human brain might have a high Kolmogorov complexity, the information for which is encoded, not in the human genome (which contains a mere 7 gigabits of information), but rather in the laws of physics, which contain arbitrarily large amounts of information, encoded in the exact values of physical constants. For example, first 30 billion decimal digits of the fine structure constant contain 100 gigabits of information, putting the genome to shame. Do I have that right? Well, I will give you points for cleverness, but I'm not buying it. I doubt that it much matters what the constants are, out past the first hundred digits or so. Yes, I realize that the details of how the universe proceeds may be chaotic; it may involve sensitive dependence both on initial conditions and on physical constants. But I don't think that really matters. Physical constants haven't changed since the Cambrian, but genomes have. And I think that it is the change in genomes which led to the human brain, the dolphin brain, the parrot brain, and the octopus brain. Alter the fine structure constant in the 2 billionth decimal place, and those brain architectures would still work, and those genomes would still specify development pathways leading to them. Or so I believe.
1timtyler
What makes you think that? ...and why not? Under the hypothesis that physics encodes relevant information, a lot of the required information was there from the beginning. The fact that brains only became manifest after the Cambrian doesn't mean the propensity for making brains was not there from the beginning. So: that observation doesn't tell you very much. Right - but what evidence do you have of that? You are aware of chaos theory, no? Small changes can lead to dramatic changes surprisingly quickly. Organisms inherit the laws of physics (and indeed the initial conditions of the universe they are in) - as well as their genomes. Information passes down the generations both ways. If you want to claim the design information is in one inheritance channel more than the other one, it seems to me that you need some evidence relating to that issue. The evidence you have presented so far seems pretty worthless - the delayed emergence of brains seems equally compatible with both of the hypotheses under consideration. So: do you have any other relevant evidence?
0WrongBot
No other rational [ETA: I meant physical and I am dumb] process is known to rely on physical constants to the degree you propose. What you propose is not impossible, but it is highly improbable.
2timtyler
What?!? What makes you think that? Sensitive dependence on initial conditions is an extremely well-known phenomenon. If you change the laws of physics a little bit, the result of a typical game of billiards will be different. This kind of phenomenon is ubiquitous in nature, from the orbit of planets, to the paths rivers take. If a butterfly's wing flap can cause a tornado, I figure a small physical constant jog could easily make the difference between intelligent life emerging, and it not doing so billions of years later. Sensitive dependence on initial conditions is literally everywhere. Check it out: http://en.wikipedia.org/wiki/Chaos_theory
1Kingreaper
Did you miss this bit: Sensitivity to initial conditions is one thing. Sensitivity to 1 billion SF in a couple of decades?
1timtyler
The universe took about 14 billion years to get this far - and if you look into the math of chaos theory, the changes propagate up very rapidly. There is an ever-expanding avalanche of changes - like an atomic explosion. For the 750mb-or-so of data under discussion, you could easily see the changes at a macroscopic scale rapidly. Atoms in stars bang into each other pretty quickly. I haven't attempted to calculate it - but probably within a few minutes, I figure.
1PaulAlmond
Would you actually go as far as maintaining that, if a change were to happen tomorrow to the 1,000th decimal place of a physical constant, it would be likely to stop brains from working, or are you just saying that a similar change to a physical constant, if it happened in the past, would have been likely to stop the sequence of events which has caused brains to come into existence?
1timtyler
Option 2. Existing brains might be OK - but I think newly-constructed ones would have to not work properly when they matured. So, option 2 would not be enough on its own.
0[anonymous]
Correction: That last line should be "which has CAUSED brains to come into existence?"
2SilasBarta
You can edit comments after submitting them -- when logged in, you should see an edit button. By the way, I'm reading your part 15, section 2 now.
0[anonymous]
Hi Silas! Thanks for telling me that. I was logged in and didn't see it, but I will look more carefully next time. I'm actually proof-reading a document now which improves the "action selection process". I was never happy with what I described and it was a kind of placeholder. The new stuff will be very short though. Anyway, what do you do? I have the idea it is something computer related, maybe?
-2[anonymous]
Apologies for the comment I inadvertently placed here. I thought I was answering a PM and did not mean to add personal exchanges. I find computers annoying sometimes, and will happily stop using them when something else that is Turing equivalent becomes available.
0JamesAndrix
First, that is VERY different than the design information being in the constant, but not in the genome. (you could more validly say that the genome is what it is because the constant is precisely what it is.) Second, the billiard ball example is invalid. It doesn't matter exactly where the billiard balls are if you're getting hustled. Neurons are not typically sensitive to the precise positions of their atoms. Information processing relies on the ability to largely overlook noise.
0WrongBot
What physical process would cease to function if you increased c by a billionth of a percent? Or one of the other Planck units? Processes involved in the functioning of both neurons and transistors don't count, because then there's no difference to account for.
2Sniffnoy
Nitpick: c is a dimensioned quantity, so changes in it aren't necessarily meaningful.
2WrongBot
*Blink.* *Reads Wikipedia.* Would I be correct in thinking that one would need to modify the relationship of c to some other constant (the physics equation that represent some physical law?) for the change to be meaningful? I may be failing to understand the idea of dimension. Thank you for the excuse to learn more math, by the way.
3Psy-Kosh
Yes, you would be correct, at least in terms of our current knowledge. In fact, it's not that unusual to choose units so that you can set c = 1 (ie, to make it unitless). This way units of time and units of distance are the same kind, velocities are dimensionless geometric quantities, etc... You might want to think of "c" not so much as a speed as a conversion factor between distance type units and time type units.
0timtyler
That isn't really the idea. It would have to interfere with the development of a baby enough for its brain not to work out properly as an adult, though - I figure.
1JamesAndrix
Artificial wombs
0RobinZ
Don't currently exist. I'm not sure that's a strong argument.
3knb
Myers has always had a tendency to attack other people's arguments like enemy soldiers. A good example is his take on evolutionary psychology, which he hates so much it is actually funny. He also claims to have desecrated a consecrated host (the sacramental wafers Catholics consider to be the body of Jesus). That will show those evil theists how a good, rational person behaves!
3Mitchell_Porter
Myers' thesis is that you are not going to figure out by brute-force physical simulation how the genome gives rise to the organism, knowing just the genomic sequence. On every scale - molecule, cell, tissue, organism - there are very complicated boundary conditions at work. You have to do experimental biology, observe those boundary conditions, and figure out what role they play. I predict he would be a lot more sympathetic if Kurzweil was talking about AIs figuring out the brain by doing experimental biology, rather than just saying genomic sequence + laws of physics will get us there.
5Perplexed
And he is quite possibly correct. However, that has nothing at all to do with what Kurzweil said. I predict he would be more sympathetic if he just made the effort to figure out what Kurzweil said. But, of course, we all know there is no chance of that, so "conjecture" might be a better word than "predict".
4Mitchell_Porter
Myers doesn't have an argument against Kurzweil's estimate of the brain's complexity. But his skepticism about Kurzweil's timescale can be expressed in terms of the difficulty of searching large spaces. Let's say it does take a million lines of code to simulate the brain. Where is the argument that we can produce the right million lines of code within twenty years? The space of million-line programs is very large.
1Perplexed
I agree, both regarding timescale, and regarding reason for timescale difficulties. As I understand Kurzweil, he is saying that we will build the AI, not by finding the program for development and simulating it, but rather by scanning the result of the development and duplicating it in a different medium. The only relevance of that hypothetical million-line program is that it effectively puts a bound on the scanning and manufacturing tolerances that we need to achieve. Well, while it is probably true in general that we don't need to get the wiring exactly right on all of the trillions of neurons, there may well be some where the exact right embryonic wiring is crucial to success. And, since we don't yet have or understand that million-line program that somehow gets the wiring right reliably, we probably won't get them right ourselves. At least not at first. It feels a little funny to find myself making here an argument right out of Bill Dembski's playbook. No free lunch! Needle in a haystack. Only way to search that space is by exhaustion. Well, we shall see what we shall see.
3SilasBarta
I agree, but at the same time, I wish biologists would learn more information theory, since their focus should be identifying the information flows going on, as this is what will lead us to a comprehensible model of human development and functionality. (I freely admit I don't have years in the trenches, so this may be a naive view, but if my experience with any other scientific turf war is any guide, this is important advice.)
2Paul Crowley
This was cited to me in a blog discussion as "schoolboy biology EY gets wrong" (he said something similar, apparently).
-2wedrifid
Personal libraries.

I upvoted this, but I'm torn about this.

In your recent posts you've been slowly, carefully, thoroughly deconstructing one person. Part of me wants to break into applause at the techniques used, and learn from them, because in my whole life of manipulation I've never mounted an attack of such scale. (The paragraph saying "something has gone very wrong" was absolutely epic, to the point of evoking musical cues somewhere at the edge of my hearing. Just like the "greatly misguided" bit in your previous post. Bravo!) But another part of me feels horror and disgust because after traumatic events in my own life I'd resolved to never do this thing again.

It comes down to this: I enjoy LW for now. If Eliezer insists on creating a sealed reality around himself, what's that to me? You don't have to slay every dragon you see. Saving one person from megalomania (real or imagined) is way less important than your own research. Imagine the worst possible world: Eliezer turns into a kook. What would that change, in the grand scheme of things or in your personal life? Are there not enough kooks in AI already?

And lastly, a note about saving people. I think many of us here have had ... (read more)

I saved someone from suicide once. While the experience was certainly quite unpleasant at the time, if I had hit "ignore," as you suggest, she would have died. I don't think that I would be better off today if I had let her die, to say nothing of her. The fact that saving people is hard doesn't mean that you shouldn't do it!

It comes down to this: I enjoy LW for now. If Eliezer insists on creating a sealed reality around himself, what's that to me? You don't have to slay every dragon you see. Saving one person from megalomania (real or imagined) is way less important than your own research. Imagine the worst possible world: Eliezer turns into a kook. What would that change, in the grand scheme of things or in your personal life?

The very fate of the universe, potentially. Purely hypothetically and for the sake of the discussion:

  • If Eliezer did have the potential to provide a strong positive influence on grand scale future outcomes but was crippled by the still hypothetical lack of self-doubt then that is a loss of real value.
  • A bad 'Frodo' can be worse than no Frodo at all. If we were to give the ring to a Frodo who thought he could take on Nazgul in hand to hand combat then we would lose the ring and so the lose the chance to give said ring to someone who could pull it off. Multi (and those for whom he asks such questions) have limited resources (and attention) so it may be worth deliberate investigation of potential recipients of trust.
  • Worse yet than a counterproductive Frodo would be a Frodo who
... (read more)

Er... I can't help but notice a certain humor in the idea that it's terrible if I'm self-deluded about my own importance because that means I might destroy the world.

7John_Baez
It's some sort of mutant version of "just because you're paranoid doesn't mean they're not out to get you".
6wedrifid
Yes, there is is a certain humor. But I hope you did read the dot points and followed the reasoning. It, among other things, suggests a potential benefit of criticism such as multi's aside from hypothetical benefits of discrediting you should it have been the case that you were not, in fact, competent.
9Perplexed
I suppose I could draw from that the inference that you have a rather inflated notion of the importance of what multi is doing here, ... but, in the immortal words of Richard Milhous Nixon, "That would be wrong." More seriously, I think everyone here realizes that EY has some rough edges, as well as some intellectual strengths. For his own self-improvement, he ought to be working on those rough edges. I suspect he is. However, in the meantime, it would be best if his responsibilities were in areas where his strengths are exploited and his rough edges don't really matter. So, just what are his current responsibilities? 1. Convincing people that UFAI constitutes a serious existential risk while not giving the whole field of futurism and existential risk reduction a bad rep. 2. Setting direction for and managing FAI and UFAI-avoidance research at SIAI. 3. Conducting FAI and UFAI-avoidance research. 4. Reviewing and doing conceptual QC on the research work product. To be honest, I don't see EY's "rough edges" as producing any problems at all with his performance on tasks #3 and #4. Only SIAI insiders know whether there has been a problem on task #2. Based on multi's arguments, I suspect he may not be doing so well on #1. So, to me, the indicated response ought to be one of the following: A. Hire someone articulate (and if possible, even charismatic) to take over task #1 and make whatever minor adjustments are needed regarding task #2. B. Do nothing. There is no problem! C. Get some academic papers published so that FAI/anti-UFAI research becomes interesting to the same funding sources that currently support CS, AI, and decision theory research. Then reconstitute SIAI as just one additional research institution which is fighting for that research funding. I would be interested in what EY thinks of these three possibilities. Perhaps for different reasons, I suspect, so would multi. [Edited to correct my hallucination of confusing multifoliaterose with wedri
1wedrifid
Was the first (unedited) 'you' intended? If so I'll note that I was merely answering a question within a counterfactual framework suggested by the context. I haven't even evaluated what potential importance multi's post may have - but the prior probability I have for 'a given post on LW mattering significantly' is not particularly high. I like your general analysis by the way and am always interested to know what the SIAI guys are doing along the lines of either your 1,2,3 or your A, B, C. I would seriously like to see C happen. Being able and willing to make that sort of move would be a huge step forward (and something that makes any hints of 'arrogance' seem trivial.)
0Unknowns
I think that originally Perplexed didn't look at your comment carefully and thought that multi had written it.
1Perplexed
Close. Actually, I had looked at the first part of the comment and then written my response under the delusion that wedrifid had been the OP. I am now going to edit my comment to cleanly replace the mistaken "you" with "multi"
1wedrifid
I think you are right. I'm just playing the disclaimer game. Since this is a political thread there is always the risk of being condemned for supporting various positions. In this case I gave a literal answer to a rhetorical question directed at multi. Following purely social reasoning that would mean that I: * Am challenging cousin_it * Condemning Eliezer * Agreeing with anything and everything said by multi and probably also with everything said by anyone else who agrees with multi. * Almost certainly saying something about the credulity of uFAI risks. * In some way think any of this is particularly important to the universe outside the time/abstract-space bubble that is LessWrong this week. Of course that comment actually lent credence to Eliezer (hence the humor) and was rather orthogonal to multi's position with respect to arrogance. It's not that I mind too much sticking my neck out risking a social thrashing here or there. It's just that I have sufficient capability for sticking my neck out for things that I actually do mean and for some reason prefer any potential criticism to be correctly targeted. It says something about many nerds that they value being comprehended more highly than approval.
0Strange7
Approval based on incomprehension is fragile and unsatisfying.
3dclayh
Veering wildly off-topic: Come on now. Humans are immortal in Tolkien, they just sit in a different waiting room. (And technically can't come back until the End of Days™, but who cares about that.)
1Strange7
Alright, then, call it her permanent resident status. If real death is off the table for everyone sapient, she's still taking as big a risk as any member of the Fellowship proper.
1dclayh
To be sure. I was only pointing out that her "giving up immortality" was not nearly as crazy as the words "giving up immortality" might suggest in other contexts.
1cousin_it
What Eliezer said. I was arguing from the assumption that he is wrong about FAI and stuff. If he's right about the object level, then he's not deluded in considering himself important.
3Vladimir_Nesov
But if he is wrong about FAI and stuff, then he is still deluded not specifically about considering himself important, that implication is correct, he is deluded about FAI and stuff.
0cousin_it
Agreed.
2wedrifid
Which, of course, would still leave the second two dot points as answers to your question.
1cousin_it
How so? Eliezer's thesis is "AGI is dangerous and FAI is possible". If he's wrong - if AGI poses no danger or FAI is impossible - then what do you need a Frodo for?
0wedrifid
Edited the grandparent to disambiguate the context. (I haven't discussed that particular thesis of Eliezer's and nor does doubting that particular belief seem to be a take home message from multi's post. The great grandparent is just a straightforward answer to the paragraph it quotes.)
3Vladimir_Nesov
The previous post was fine, but this one is sloppy, and I don't think it's some kind of Machiavellian plot.
2xamdam
Because you were on the giving or on the receiving end of it? Agreed; personally I de-converted myself from orthodox judaism, but I still find it crazy when people write big scholarly books debunking the bible; it's just useless a waste of energy (part of it is academic incentives). I haven't been involved in these situations, but taking a cue from drug addicts (who incidentally have high suicide rate) most of them do not recover, but maybe 10% do. So most of the time you'll find frustration, but one in 10 you'd save a life, I am not sure if that's worthless.

It seems like an implication of your post that no one is ever allowed to believe they're saving the world. Do you agree that this is an implication? If not, why not?

Not speaking for multi, but, in any x-risk item (blowing up asteroids, stabilizing nuclear powers, global warming, catastrophic viral outbreak, climate change of whatever sort, FAI, whatever) for those working on the problem, there are degrees of realism:

"I am working on a project that may have massive effect on future society. While the chance that I specifically am a key person on the project are remote, given the fine minds at (Google/CDC/CIA/whatever), I still might be, and that's worth doing." - Probably sane, even if misguided.

"I am working on a project that may have massive effect on future society. I am the greatest mind in the field. Still, many other smart people are involved. The specific risk I am worried about may or not occur, but efforts to prevent its occurrence are valuable. There is some real possibility that I will the critical person on the project." - Possibly sane, even if misguided.

"I am working on a project that will save a near-infinite number of universes. In all likelihood, only I can achieve it. All of the people - even people perceived as having better credentials, intelligence, and ability - cannot do what I am doing. All crit... (read more)

[-][anonymous]140

I don't think Eliezer believes he's irreplaceable, exactly. He thinks, or I think he thinks, that any sufficiently intelligent AI which has not been built to the standard of Friendliness (as he defines it) is an existential risk. And the only practical means for preventing the development of UnFriendly AI is to develop superintelligent FAI first. The team to develop FAI needn't be SIAI, and Eliezer wouldn't necessarily be the most important contributor to the project, and SIAI might not ultimately be equal to the task. But if he's right about the risk and the solution, and his untimely demise were to doom the world, it would be because no-one else tried to do this, not because he was the only one who could.

Not that this rules out your interpretation. I'm sure he has a high opinion of his abilities as well. Any accusation of hubris should probably mention that he once told Aubrey de Grey "I bet I can solve ALL of Earth's emergency problems before you cure aging."

3JamesAndrix
There may be multiple different projects projects, each necessary to save the world, and each having a key person who knows more about the project, and/or is more driven and/or is more capable than anyone else. Each such person has weirdly high expected utility, and could accurately make a statement like EY's and still not be the person with the highest expected utility. Their actual expected utility would depend on the complexity of the project and the surrounding community, and how much the success of the project alters the value of human survival. This is similar to the idea that responsibility is not a division of 100%. http://www.ranprieur.com/essays/mathres.html
3Jonathan_Graehl
What you say sounds reasonable, but I feel it's unwise for me to worry about such things. If I were to sound such a vague alarm, I wouldn't expect anyone to listen to me unless I'd made significant contributions in the field myself (I haven't).
5Unknowns
Multifoliaterose said this: Note that there are qualifications on this. If you're standing by the button that ends the world, and refuse to press it when urged, or you prevent others from pressing it (e.g. Stanislav Petrov), then you may reasonably believe that you're saving the world. But no, you may not reasonably believe that you are saving the world based on long chains of reasoning based on your intuition, not on anything as certain as mathematics and logic, especially decades in advance of anything happening.
2Eliezer Yudkowsky
It seems like an implication of this and other assumptions made by multi, and apparently shared by you, is that no one can believe themselves to be critical to a Friendly AI project that has a significant chance of success. Do you agree that this is an implication? If not, why not?
8Unknowns
No, I don't agree this is an implication. I would say that no one can reasonably believe all of the following at the same time with a high degree of confidence: 1) I am critical to this Friendly AI project that has a significant chance of success. 2) There is no significant chance of Friendly AI without this project. 3) Without Friendly AI, the world is doomed. But then, as you know, I don't consider it reasonable to put a high degree in confidence in number 3. Nor do many other intelligent people (such as Robin Hanson.) So it isn't surprising that I would consider it unreasonable to be sure of all three of them. I also agree with Tetronian's points.
3Eliezer Yudkowsky
I see. So it's not that any one of these statements is a forbidden premise, but that their combination leads to a forbidden conclusion. Would you agree with the previous sentence? BTW, nobody please vote down the parent below -2, that will make it invisible. Also it doesn't particularly deserve downvoting IMO.
7Perplexed
I would suggest that, in order for this set of beliefs to become (psychiatrically?) forbidden, we need to add a fourth item. 4) Dozens of other smart people agree with me on #3. If someone believes that very, very few people yet recognize the importance of FAI, then the conjunction of beliefs #1 thru #3 might be reasonable. But after #4 becomes true (and known to our protagonist), then continuing to hold #1 and #2 may be indicative of a problem.
7Eliezer Yudkowsky
Dozens isn't sufficient. I asked Marcello if he'd run into anyone who seemed to have more raw intellectual horsepower than me, and he said that John Conway gave him that impression. So there are smarter people than me upon the Earth, which doesn't surprise me at all, but it might take a wider net than "dozens of other smart people" before someone comes in with more brilliance and a better starting math education and renders me obsolete.
9Spurlock
Simply out of curiosity: Plenty of criticism (some of it reasonable) has been lobbed at IQ tests and at things like the SAT. Is there a method known to you (or anyone reading) that actually measures "raw intellectual horsepower" in a reliable and accurate way? Aside from asking Marcello.

Aside from asking Marcello.

I was beginning to wonder if he's available for consultation.

8rabidchicken
Read the source code, and then visualize a few levels from Crysis or Metro 2033 in your head. While you render it, count the average Frames per second. Alternatively, see how quickly you can find the prime factors of every integer from 1 to 1000. Which is to say... Humans in general have extremely limited intellectual power. instead of calculating things efficiently, we work by using various tricks with caches and memory to find answers. Therefore, almost all tasks are more dependant on practice and interest than they are on intelligence. So, rather then testing the statement "Eliezer is smart" it has more bearing on this debate to confirm "Eliezer has spent a large amount of time optimizing his cache for tasks relating to rationality, evolution, and artificial intelligence". Intelligence is overrated.
1XiXiDu
Sheer curiosity, but have you or anyone ever contacted John Conway about the topic of u/FAI and asked him what the thinks about the topic, the risks associated with it and maybe the SIAI itself?
1xamdam
"raw intellectual power" != "relevant knowledge". Looks like he worked on some game theory, but otherwise not much relevancy. Should we ask Steven Hawking? Or take a poll of Nobel Laureates? I am not saying that he can't be brought up to date in this kind of discussion, and has a lot to consider, but not asking him as things are indicates little.
0XiXiDu
Richard Dawkins seems to have enough power to infer the relevant knowledge from a single question.
1Perplexed
Candid, and fair enough.
0whowhowho
Raw intellectual horsepower is not the right kind of smart.
-2TheAncientGeek
Domain knowledge is much more relevant than raw intelligence.
2Perplexed
With the hint from EY on another branch, I see a problem in my argument. Our protagonist might circumvent my straitjacket by also believing 5) The key to FAI is TDT, but I have been so far unsuccessful in getting many of those dozens of smart people to listen to me on that subject. I now withdraw from this conversation with my tail between my legs.
2katydee
All this talk of "our protagonist," as well the weird references to SquareSoft games, is very off-putting for me.
4Unknowns
I wouldn't put it in terms of forbidden premises or forbidden conclusions. But if each of these statements has a 90% of being true, and if they are assumed to be independent (which admittedly won't be exactly true), then the probability that all three are true would be only about 70%, which is not an extremely high degree of confidence; more like saying, "This is my opinion but I could easily be wrong." Personally I don't think 1) or 3), taken in a strict way, could reasonably be said to have more than a 20% chance of being true. I do think a probability of 90% is a fairly reasonable assignment for 2), because most people are not going to bother about Friendliness. Accounting for the fact that these are not totally independent, I don't consider a probability assignment of more than 5% for the conjunction to be reasonable. However, since there are other points of view, I could accept that someone might assign the conjunction a 70% chance in accordance with the previous paragraph, without being crazy. But if you assign a probability much more than that I would have to withdraw this. If the statements are weakened as Carl Shulman suggests, then even the conjunction could reasonably be given a much higher probability. Also, as long as it is admitted that the probability is not high, you could still say that the possibility needs to be taken seriously because you are talking about the possible (if yet improbable) destruction of the world.

I certainly do not assign a probability as high as 70% to the conjunction of all three of those statements.

And in case it wasn't clear, the problem I was trying to point out was simply with having forbidden conclusions - not forbidden by observation per se, but forbidden by forbidden psychology - and using that to make deductions about empirical premises that ought simply to be evaluated by themselves.

I s'pose I might be crazy, but you all are putting your craziness right up front. You can't extract milk from a stone!

3PaulAlmond
Just curious (and not being 100% serious here): Would you have any concerns about the following argument (and I am not saying I accept it)? 1. Assume that famous people will get recreated as AIs in simulations a lot in the future. School projects, entertainment, historical research, interactive museum exhibits, idols to be worshipped by cults built up around them, etc. 2. If you save the world, you will be about the most famous person ever in the future. 3. Therefore there will be a lot of Eliezer Yudkowsky AIs created in the future. 4. Therefore the chances of anyone who thinks he is Eliezer Yudkowsky actually being the orginal, 21st century one are very small. 5. Therefore you are almost certainly an AI, and none of the rest of us are here - except maybe as stage props with varying degrees of cognition (and you probably never even heard of me before, so someone like me would probably not get represented in any detail in an Eliezer Yudkowsky simulation). That would mean that I am not even conscious and am just some simple subroutine. Actually, now I have raised the issue to be scary, it looks a lot more alarming for me than it does for you as I may have just argued myself out of existence...
3wedrifid
That doesn't seem scary to me at all. I still know that there is at least one of me that I can consider 'real'. I will continue to act as if I am one of the instances that I consider me/important. I've lost no existence whatsoever.
0Wei Dai
You can see Eliezer's position on the Simulation Argument here.
3Unknowns
That's good to know. I hope multifoliaterose reads this comment, as he seemed to think that you would assign a very high probability to the conjunction (and it's true that you've sometimes given that impression by your way of talking.) Also, I didn't think he was necessarily setting up forbidden conclusions, since he did add some qualifications allowing that in some circumstances it could be justified to hold such opinions.
-3multifoliaterose
To be quite clear about which of Unknowns' points I object, my main objection is to the point: where 'I' is replaced by "Eliezer." I assign a probability of less than 10^(-9) to you succeeding in playing a critical role on the Friendly AI project that you're working on. (Maybe even much less than that - I would have to spend some time calibrating my estimate to make a judgment on precisely how low a probability I assign to the proposition.) My impression is that you've greatly underestimated the difficulty of building a Friendly AI.

I assign a probability of less than 10^(-9) to you succeeding in playing a critical role on the Friendly AI project that you're working on.

I wish the laws of argument permitted me to declare that you had blown yourself up at this point, and that I could take my toys and go home. Alas, arguments are not won on a points system.

My impression is that you've greatly underestimated the difficulty of building a Friendly AI.

Out of weary curiosity, what is it that you think you know about Friendly AI that I don't?

And has it occurred to you that if I have different non-crazy beliefs about Friendly AI then my final conclusions might not be so crazy either, no matter what patterns they match in your craziness recognition systems?

I agree it's kind of ironic that multi has such an overconfident probability assignment right after criticizing you for being overconfident. I was quite disappointed with his response here.

2multifoliaterose
Why does my probability estimate look overconfident?

One could offer many crude back-of-envelope probability calculations. Here's one: let's say there's

  • a 10% chance AGI is easy enough for the world to do in the next few decades
  • a 1% chance that if the world can do it, a team of supergeniuses can do the Friendly kind first
  • an independent 10% chance Eliezer succeeds at putting together such a team of supergeniuses

That seems conservative to me and implies at least a 1 in 10^4 chance. Obviously there's lots of room for quibbling here, but it's hard for me to see how such quibbling could account for five orders of magnitude. And even if post-quibbling you think you have a better model that does imply 1 in 10^9, you only need to put little probability mass on my model or models like it for them to dominate the calculation. (E.g., a 9 in 10 chance of a 1 in 10^9 chance plus a 1 in 10 chance of a 1 in 10^4 chance is close to a 1 in 10^5 chance.)

0multifoliaterose
I don't find these remarks compelling. I feel similar remarks could be used to justify nearly anything. Of course, I owe you an explanation. One will follow later on.
3Unknowns
Unless you've actually calculated the probability mathematically, a probability of one in a billion for a natural language claim that a significant number of people accept as likely true is always overconfident. Even Eliezer said that he couldn't assign a probability as low as one in a billion for the claim "God exists" (although Michael Vassar criticized him for this, showing himself to be even more overconfident than Eliezer.)
7komponisto
I'm afraid I have to take severe exception to this statement. You give the human species far too much credit if you think that our mere ability to dream up a hypothesis automatically raises its probability above some uniform lower bound.
1Unknowns
I am aware of your disagreement, for example as expressed by the absurd claims here. Yes, my basic idea is, unlike you, to give some credit to the human species. I think there's a limit on how much you can disagree with other human beings-- unless you're claiming to be something superhuman. Did you see the link to this comment thread? I would like to see your response to the discussion there.
7komponisto
At least for epistemic meanings of "superhuman", that's pretty much the whole purpose of LW, isn't it? My immediate response is as follows: yes, dependency relations might concentrate most of the improbability of a religion to a relatively small subset of its claims. But the point is that those claims themselves possess enormous complexity (which may not necessarily be apparent on the surface; cf. the simple-sounding "the woman across the street is a witch; she did it").

Let's pick an example. How probable do you think it is that Islam is a true religion? (There are several ways to take care of logical contradictions here, so saying 0% is not an option.)

Suppose there were a machine--for the sake of tradition, we can call it Omega--that prints out a series of zeros and ones according to the following rule. If Islam is true, it prints out a 1 on each round, with 100% probability. If Islam is false, it prints out a 0 or a 1, each with 50% probability.

Let's run the machine... suppose on the first round, it prints out a 1. Then another. Then another. Then another... and so on... it's printed out 10 1's now. Of course, this isn't so improbable. After all, there was a 1/1024 chance of it doing this anyway, even if Islam is false. And presumably we think Islam is more likely than this to be false, so there's a good chance we'll see a 0 in the next round or two...

But it prints out another 1. Then another. Then another... and so on... It's printed out 20 of them. Incredible! But we're still holding out. After all, million to one chances happen every day...

Then it prints out another, and another... it just keeps going... It's printed out 30 1's now. Of course... (read more)

Thank you a lot for posting this scenario. It's instructive from the "heuristics and biases" point of view.

Imagine there are a trillion variants of Islam, differing by one paragraph in the holy book or something. At most one of them can be true. You pick one variant at random, test it with your machine and get 30 1's in a row. Now you should be damn convinced that you picked the true one, right? Wrong. Getting this result by a fluke is 1000x more likely than having picked the true variant in the first place. Probability is unintuitive and our brains are mush, that's all I'm sayin'.

1Unknowns
I agree with this. But if the scenario happened in real life, you would not be picking a certain variant. You would be asking the vague question, "Is Islam true," to which the answer would be yes if any one of those trillion variants, or many others, were true. Yes, there are trillions of possible religions that differ from one another as much as Islam differs from Judaism, or whatever. But only a few of these are believed by human beings. So I still think I would convert after 30 1's, and I think this would reasonable.
7cousin_it
If a religion's popularity raises your prior for it so much, how do you avoid Pascal's Mugging with respect to the major religions of today? Eternity in hell is more than 2^30 times worse than anything you could experience here; why aren't you religious already?
3Unknowns
It doesn't matter whether it raises your prior or not; eternity in hell is also more than 2^3000 times worse etc... so the same problem will apply in any case. Elsewhere I've defended Pascal's Wager against the usual criticisms, and I still say it's valid given the premises. But there are two problematic premises: 1) It assumes that utility functions are unbounded. This is certainly false for all human beings in terms of revealed preference; it is likely false even in principle (e.g. the Lifespan Dilemma). 2) It assumes that humans are utility maximizers. This is false in fact, and even in theory most of us would not want to self-modify to become utility maximizers; it would be a lot like self-modifying to become a Babyeater or a Super-Happy.
1Wei Dai
Do you have an answer for how to avoid giving in to the mugger in Eliezer's original Pascal's Mugging scenario? If not, I don't think your question is a fair one (assuming it's meant to be rhetorical).
0cousin_it
I don't have a conclusive answer, but many people say they have bounded utility functions (you see Unknowns pointed out that possibility too). The problem with assigning higher credence to popular religions is that it forces your utility bound to be lower if you want to reject the mugging. Imagining a billion lifetimes is way easier than imagining 3^^^^3 lifetimes. That was the reason for my question.
2Wei Dai
My answer (for why I don't believe in a popular religion as a form of giving in to a Pascal's Mugging) would be that I'm simultaneously faced with a number of different Pascal's Muggings, some of which are mutually exclusive, so I can't just choose to give in to all of them. And I'm also unsure of what decision theory/prior/utility function I should use to decide what to do in the face of such Muggings. Irreversibly accepting any particular Mugging in my current confused state is likely to be suboptimal, so the best way forward at this point seems to be to work on the relevant philosophical questions.
0endoself
That's what I think too! You're only the second other person I have seen make this explicit, so I wonder how many people have even considered this. Do you think more people would benefit from hearing this argument?
0Wei Dai
Sure, why do you ask? (If you're asking because I've thought of this argument but haven't already tried to share it with a wider audience, it probably has to do with reasons, e.g., laziness, that are unrelated to whether I think more people would benefit from hearing it.)
0endoself
I was considering doing a post on it, but there are many posts that I want to write, many of which require research, so I avoided implying that it would be done soon/ever.
1thomblake
Oddly, I think you meant "Pascal's Wager".
2FAWS
Pascal's Mugging. Pascal's Wager with something breaking symmetry (in this case observed belief of others).
0thomblake
Yes, I suppose it is technically a Pascal's Mugging. I think Pascal thought he was playing Pascal's Mugging though.
2FAWS
I don't think Pascal recognized any potential symmetry in the first place, or he would have addressed it properly.
0Vladimir_Nesov
Privileging the hypothesis! That they are believed by human beings doesn't lend them probability.
4FAWS
Well, it does to the extent that lack of believers would be evidence against them. I'd say that Allah is considerably more probable than a similarly complex and powerful god who also wants to be worshiped and is equally willing to interact with humans, but not believed in by anyone at all. Still considerably less probable than the prior of some god of that general sort existing, though.
0Vladimir_Nesov
Agreed, but then we have the original situation, if we only consider the set of possible gods that have the property of causing worshiping of themselves.
2Unknowns
This whole discussion is about this very point. Downvoted for contradicting my position without making an argument.
1Vladimir_Nesov
Your position statement didn't include an argument either, and the problem with it seems rather straightforward, so I named it.
0Unknowns
I've been arguing with Sewing Machine about it all along.
2Perplexed
No. It doesn't lend probability, but it seems like it ought to lend something. What is this mysterious something? Lets call it respect. Privileging the hypothesis is a fallacy. Respecting the hypothesis is a (relatively minor) method of rationality. We respect the hypotheses that we find in a math text by investing the necessary mental resources toward the task of finding an analytic proof. We don't just accept the truth of the hypothesis on authority. But on the other hand, we don't try to prove (or disprove) just any old hypothesis. It has to be one that we respect. We respect scientific hypotheses enough to invest physical resources toward performing experiments that might refute or confirm them. We don't expend those resources on just any scientific hypothesis. Only the ones we respect. Does a religion deserve respect because it has believers? More respect if it has lots of believers? I think it does. Not privilege. Definitely not. But respect? Why not?
5FAWS
You can dispense with this particular concept of respect since in both your examples you are actually supplied with sufficient Bayesian evidence to justify evaluating the hypothesis, so it isn't privileged. Whether this is also the case for believed in religions is the very point contested.
5Vladimir_Nesov
No, it's a method of anti-epistemic horror.
0[anonymous]
Yes, this seems right. A priori, with no other evidence one way or another, a belief held by human beings is more likely to be true than not. If Ann says she had a sandwich for lunch, then her words are evidence that she actually had a sandwich for lunch. Of course, we have external reason to doubt lots of things that human beings claim and believe, including religions. And a religion does not become twice as credible if it has twice as many adherents. Right now I believe we have good reason to reject (at least some of) the tenets of all religious traditions. But it does make some sense to give some marginal privilege or respect to an idea based on the fact that somebody believes it, and to give the idea more credit if it's very durable over time, or if particularly clever people believe it. If it were any subject but religion -- if it were science, for instance -- this would be an obvious point. Scientific beliefs have often been wrong, but you'll be best off giving higher priors to hypotheses believed by scientists than to other conceivable hypotheses.
1Unknowns
Also... if you haven't been to Australia, is it privileging the hypothesis to accept the word of those who say that it exists? There are trillions of possible countries that could exist that people don't believe exist... And don't tell me they say they've been there... religious people say they've experienced angels etc. too. And so on. People's beliefs in religion may be weaker than their belief in Austrialia, but it certainly is not privileging a random hypothesis.
1Vladimir_Nesov
Your observations (of people claiming to having seen an angel, or a kangaroo) are distinct from hypotheses formed to explain those observations. If in a given case, you don't have reason to expect statements people make to be related to facts, then the statements people make taken verbatim have no special place as hypotheses.
0Unknowns
"You don't have reason to expect statements people make to be related to facts" doesn't mean that you have 100% certainty that they are not, which you would need in order to invoke privileging the hypothesis.
3[anonymous]
Why do you have at most 99.999999999% certainty that they are not? Where does that number one-minus-a-billionth come from?
0Unknowns
The burden of proof is on the one claiming a greater certainty (although I will justify this later in any case.)
1Vladimir_Nesov
Now you are appealing to impossibility of absolute certainty, refuting my argument as not being that particular kind of proof. If hypothesis X is a little bit more probable than many others, you still don't have any reason to focus on it (and correlation could be negative!).
-2Unknowns
In principle the correlation could be negative but this is extremely unlikely and requires some very strange conditions (for example if the person is more likely to say that Islam is true if he knows it is false than if he knows it is true).
1thomblake
Begging the question!
0Cyan
I disagree; given that most of the religions in question center on human worship of the divine, I have to think that Pr(religion X becomes known among humans | religion X is true) > Pr(religion X does not become known among humans | religion X is true). But I hate to spend time arguing about whether a likelihood ratio should be considered strictly equal to 1 or equal to 1 + epsilon when the prior probabilities of the hypotheses in question are themselves ridiculously small.
5komponisto
Of course I'm serious (and I hardly need to point out the inadequacy of the argument from the incredulous stare). If I'm not going to take my model of the world seriously, then it wasn't actually my model to begin with. Sewing-Machine's comment below basically reflects my view, except for the doubts about numbers as a representation of beliefs. What this ultimately comes down to is that you are using a model of the universe according to which the beliefs of Muslims are entangled with reality to a vastly greater degree than on my model. Modulo the obvious issues about setting up an experiment like the one you describe in a universe that works the way I think it does, I really don't have a problem waiting for 66 or more 1's before converting to Islam. Honest. If I did, it would mean I had a different understanding of the causal structure of the universe than I do. Further below you say this, which I find revealing: As it happens, given my own particular personality, I'd probably be terrified. The voice in my head would be screaming. In fact, at that point I might even be tempted to conclude that expected utilities favor conversion, given the particular nature of Islam. But from an epistemic point of view, this doesn't actually change anything. As I argued in Advancing Certainty, there is such a thing as epistemically shutting up and multiplying. Bayes' Theorem says the updated probability is one in a hundred billion, my emotions notwithstanding. This is precisely the kind of thing we have to learn to do in order to escape the low-Earth orbit of our primitive evolved epistemology -- our entire project here, mind you -- which, unlike you (it appears), I actually believe is possible.
5Wei Dai
Has anyone done a "shut up and multiply" for Islam (or Christianity)? I would be interested in seeing such a calculation. (I did a Google search and couldn't find anything directly relevant.) Here's my own attempt, which doesn't get very far. Let H = "Islam is true" and E = everything we've observed about the universe so far. According to Bayes: P(H | E) = P(E | H) P(H) / P(E) Unfortunately I have no idea how to compute the terms above. Nor do I know how to argue that P(H|E) is as small as 10^-20 without explicitly calculating the terms. One argument might be that P(H) is very small because of the high complexity of Islam, but since E includes "23% of humanity believe in some form of Islam", the term for the complexity of Islam seems to be present in both the numerator and denominator and therefore cancel each other out. If someone has done such a calculation/argument before, please post a link?
6cousin_it
P(E) includes the convincingness of Islam to people on average, not the complexity of Islam. These things are very different because of the conjunction fallacy. So P(H) can be a lot smaller than P(E).
4Wei Dai
I don't understand how P(E) does not include a term for the complexity of Islam, given that E contains Islam, and E is not so large that it takes a huge number of bits to locate Islam inside E.
1Vladimir_Nesov
It doesn't take a lot of bits to locate "Islam is false" based on "Islam is true". Does it mean that all complex statements have about 50% probability?
0cousin_it
I just wrote a post about that.
1Furcas
I don't think that's true; cousin_it had it right the first time. The complexity of Islam is the complexity of a reality that contains an omnipotent creator, his angels, Paradise, Hell, and so forth. Everything we've observed about the universe includes people believing in Islam, but not the beings and places that Islam says exist. In other words, E contains Islam the religion, not Islam the reality.
2PaulAlmond
The really big problem with such a reality is that it contains a fundamental, non-contingent mind (God's/Allah's, etc) - and we all know how much describing one of those takes - and the requirement that God is non-contingent means we can't use any simpler, underlying ideas like Darwinian evolution. Non-contingency, in theory selection terms, is a god killer: It forces God to incur a huge information penalty - unless the theist refuses even to play by these rules and thinks God is above all that - in which case they aren't even playing the theory selection game.
2Perplexed
I don't see this. Why assume that the non-contingent, pre-existing God is particularly complex. Why not assume that the current complexity of God (if He actually is complex) developed over time as the universe evolved since the big bang. Or, just as good, assume that God became complex before He created this universe. It is not as if we know enough about God to actually start writing down that presumptive long bit string. And, after all, we don't ask the big bang to explain the coastline of Great Britain.
1PaulAlmond
If we do that, should we even call that "less complex earlier version of God" God? Would it deserve the title?
1Perplexed
Sure, why not? I refer to the earlier, less complex version of Michael Jackson as "Michael Jackson".
1Furcas
Agreed. It's why I'm so annoyed when even smart atheists say that God was an ok hypothesis before evolution was discovered. God was always one of the worst possible hypotheses! Or, put more directly: Unless the theist is deluding himself. :)
2cousin_it
I'm confused. In the comments to my post you draw a distinction between an "event" and a "huge set of events", saying that complexity only applies to the former but not the latter. But Islam is also a "huge set of events" - it doesn't predict just one possible future, but a wide class of them (possibly even including our actual world, ask any Muslim!), so you can't make an argument against it based on complexity of description alone. Does this mean you tripped on the exact same mine I was trying to defuse with my post? I'd be very interested in hearing a valid argument about the "right" prior we should assign to Islam being true - how "wide" the set of world-programs corresponding to it actually is - because I tried to solve this problem and failed.
0[anonymous]
Sorry, I was confused. Just ignore that comment of mine in your thread. I'm not sure how to answer your question because as far as I can tell you've already done so. The complexity of a world-program gives its a priori probability. The a priori probability of a hypothesis is the sum of the probabilities of all the world-programs it contains. What's the problem?
0byrnema
The problem is that reality itself is apparently fundamentally non-contingent. Adding "mind" to all that doesn't seem so unreasonable.
0PaulAlmond
Do you mean it doesn't seem so unreasonable to you, or to other people?