Facing the Intelligence Explosion discussion page

by lukeprog1 min read26th Nov 2011141 comments


Intelligence Explosion
Personal Blog

I've created a new website for my ebook Facing the Intelligence Explosion:


Sometime this century, machines will surpass human levels of intelligence and ability, and the human era will be over. This will be the most important event in Earth’s history, and navigating it wisely may be the most important thing we can ever do.

Luminaries from Alan Turing and Jack Good to Bill Joy and Stephen Hawking have warned us about this. Why do I think they’re right, and what can we do about it?

Facing the Intelligence Explosion is my attempt to answer those questions.



This page is the dedicated discussion page for Facing the Intelligence Explosion.

If you'd like to comment on a particular chapter, please give the chapter name at top of your comment so that others can more easily understand your comment. For example:

Re: From Skepticism to Technical Rationality

Here, Luke neglects to mention that...

137 comments, sorted by Highlighting new comments since Today at 12:05 AM
New Comment
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

Re: this image

Fucking brilliant.

I was relatively happy with that one. It's awfully hard to represent the Singularity visually.

Uncropped, in color, for your pleasure.

2Vladimir_Nesov9yWhere is it from?
5Dufaer9yFrom the "About" page [http://facingthesingularity.com/about/]:
0katydee9yYeah, that's a great image. In my opinion, it would be slightly improved if some of the original fog was drifting in front of/around the towers of the city, as this would both be a nice callback to the original art and help show that the future is still somewhat unclear, but the effort involved might be incommensurate with the results.
2lukeprog9yI would love to have a digital artist draw something very much like my mashup (but incorporating your suggestion) as an original composition, so I could use it.
1Paul Crowley8yI note that since this was written you've had a new, closely related image redrawn by Eran Cantrell. I'd actually thought it was Aaron Diaz from the way the guy looks...
0kurzninja8yLooking at the new image now that I've just finished reading the E-book, I could have sworn it was Benedict Cumberbatch playing Sherlock Holmes.
0loup-vaillant9yYeah, except the minor quibble [http://lesswrong.com/lw/9jr/could_democritus_have_predicted_intelligence/5qt4] I have over what this image actually represents if you have some background knowledge (the space city is from a computer game). But sure, the look of it is absolutely brilliant.
0Bongo9yIt's also another [http://lesswrong.com/lw/8f0/existential_risk/59ee] far-mode picture.
[-][anonymous]9y 11

I find this chatty, informal style a lot easier to read than your formal style. The sentences are shorter and easier to follow, and it flows a lot more naturally. For example, compare your introduction to rationality in From Skepticism to Technical Rationality to that in The Cognitive Science of Rationality. Though the latter defines its terms more precisely, the former is much easier to read.

Probably relevant: this comment by Yvain.

Re: Preface

Luke discusses his conversion from Christianity to atheism in the preface. This journey plays a big role in how he came to be interested in the Singularity, but this conversion story might mistakenly lead readers to think that the arguments in favor of believing in a Singularity can also be used to argue against the existence of a God. If you want to get people to think rationally about the future of machine intelligence you might not want to intertwine your discussion with religion.

5Normal_Anomaly9yI think the target audience mostly consists of atheists, to the point where associating Singularitarianism with Atheism will help more than hurt. Especially because "it's like a religion" is the most common criticism of the Singularity idea. On another note, that paragraph has a typo:

Especially because "it's like a religion" is the most common criticism of the Singularity idea.

Which is exactly why I worry about a piece that might be read as "I used to have traditional religion. Now I have Singularitarianism!"

0Normal_Anomaly9yYes, that is a problem. I was responding to: which I saw as meaning that Singularitarianism might be perceived as associated with atheism. Associating it with atheism would IMO be a good thing, because it's already associated with religion like you said. The question is, does the page as currently written cause people to connect Singularitarianism with religion or atheism?
3Logos019yNot if it causes people to associate Singularitarianism as a "religion-substitute for atheists".
3lukeprog9yFixed, thanks.
4Vladimir_Nesov9ySome of them surely can.
5shminux9yname three
  • Kolmogorov Complexity/Occam's Razor

  • Reductionism/Physicialism

  • Rampant human cognitive biases

0[anonymous]9y(1) If the Singularity leads to the destruction of homo sapiens sapiens as a species and a physically and functionally different set of organisms (virtual or material) appears that can comprehend and imagine everything humans could and much much more (or perhaps can't imagine some!), and this claim can be made fairly definitively, I claim it updates "God made mankind in his image" to negligible probability. (2) If theoretical physicists come to a concensus in the ontological context of Penrose’s math-matter-mind triangle (see http://arxiv.org/pdf/physics/0510188v2 [http://arxiv.org/pdf/physics/0510188v2]), such as for example finding a way to agree on the "?" in the mystic approach, and such a consensus requires post-Singularity minds, the question of existence of God dissolves. (3) In the hypothetical case that (i) the picture of humanity as a species arising out of natural selection due to competition between genes is accurate, (ii) we rebel against our genes (see http://hanson.gmu.edu/matrix.html [http://hanson.gmu.edu/matrix.html] ) and the physical structure of DNA no longer is present on the planet or is somehow fundamentally altered, (iii) the mental concept of "God" is a corollary of particular gene expression, then nobody will be talking or thinking about "the existence of a God", and in some sense "God" will cease to exist as a serious idea and become a cute story, in analogy to alchemy, or demons inducing madness. You can still talk about turning iron into gold, or someone in an epileptic bout being controlled by a demon, but the mainstream of conscious organisms no longer have belief in belief or belief in that sort of talk. A modern medical understanding of certain psychological disorders is functionally equivalent to an argument that demons don't exist. All three examples share a key feature: in some sense, a Singularity might brainwash people (or the information processes descended from people) into believing God does not exist in the same sense t
1Kevin9yI think it's possible that the paragraph could be tweaked in such a way as to make religious people empathize with Luke's religious upbringing rather than be alienated by it.
3lukeprog9yAny suggestions for how?

Re: Preface

spelling: immanent -> imminent

4lukeprog9yNice. I have definitely spelled that word incorrectly every single time throughout my entire life, since the other spelling is a different word and thus not caught by spell-checkers. Thanks!

I just wanted to say the French translation is of excellent quality. Whoever is writing it, thanks for that. It helps me learn the vocabulary so I can have better discussions with French-speaking people.

Re: Superstition in Retreat

I think the cartoon is used effectively.

My one criticism is that it's not clear what the word "must" means in the final paragraph. (AI is necessary for the progress of science? AI is a necessary outcome of the progress of science? Creation of AI is a moral imperative?) Your readers may already have encountered anti-Singularitarian writings which claim that Singularitarianism conflates is with ought.

Re: Playing Taboo with "Intelligence"

Jaan Tallinn said to me:

i use "given initial resources" instead of "resources used" -- since resource acquisition is an instrumental goal, the future availability of resources is a function of optimisation power...

Fair enough. I won't add this qualification to the post because it breaks the flow and I think I can clarify when this issue comes up, but I'll note the qualification here, in the comments.

Re: Preface and Contents

I am somewhat irked by "the human era will be over" phrase. It is not given that current-type humans cease to exist after any Singularity. Also, a positive Singularity could be characterised as the beginning of the humane era, in which case it is somewhat inappropriate to refer to the era afterwards as non-human. In contrast to that, negative Singularities typically result in universes devoid of human-related things.

2Logos019yIron still exists, and is actively used, but we no longer live in the Iron Age.
0Kutta9yMy central point is contained in the sentence after that. A positive Singularity seems extremely human to me when contrasted to paperclip Singularities.
-1Logos019yI am not particularly in the habit of describing human beings as "humane". That is a trait of those things which we aspire to but only too-rarely achieve.

in http://intelligenceexplosion.com/2012/engineering-utopia/ you say "There was once a time when the average human couldn’t expect to live much past age thirty."

this is false, right?

(edit note: life expectancy matches "what the average human can expect to live to" now somewhat, but if you have a double hump of death at infancy/childhood and then old age, you can have a life expectancy of 30 but a life expectancy of 15 year olds of 60, in which case the average human can expect to live to 1 or 60 (this is very different from "can't ... (read more)

0CarlShulman7yLife expectancy used to be very low, but it was driven by child and infant mortality more than later pestilence and the like.
0alicey7yhave edited original comment to address this. (thought it was obvious)
0[anonymous]7yNo (it was still in the 30's in some parts of the world as recently as the 20th century).
0alicey7yhave edited original comment . does it address this?
0[anonymous]7yNo. Still throughout most history it was the exception to live much longer than child bearing age (14-30).
1alicey7yi can find many sources claiming the opposite (for example, http://books.google.com/books?id=EFI7tr9XK6EC&pg=PA62&dq=life+expectancy+ancient+rome+infant+mortality&hl=en&sa=X&ei=zniRUKH5Ae--2AW6sIDICQ&ved=0CEEQ6AEwBw#v=onepage&q=life%20expectancy%20ancient%20rome%20infant%20mortality&f=false [http://books.google.com/books?id=EFI7tr9XK6EC&pg=PA62&dq=life+expectancy+ancient+rome+infant+mortality&hl=en&sa=X&ei=zniRUKH5Ae--2AW6sIDICQ&ved=0CEEQ6AEwBw#v=onepage&q=life%20expectancy%20ancient%20rome%20infant%20mortality&f=false] http://books.google.com/books?id=zlsaLgBdLF8C&pg=PA44&dq=ancient+world+life+expectancy+compared&hl=en&sa=X&ei=JhyEUaKhEqmiiQKZ0YCoBQ&ved=0CDoQ6AEwAQ#v=onepage&q=ancient%20world%20life%20expectancy%20compared&f=false [http://books.google.com/books?id=zlsaLgBdLF8C&pg=PA44&dq=ancient+world+life+expectancy+compared&hl=en&sa=X&ei=JhyEUaKhEqmiiQKZ0YCoBQ&ved=0CDoQ6AEwAQ#v=onepage&q=ancient%20world%20life%20expectancy%20compared&f=false] ) and few agreeing with you cite?

Facing the Intelligence Explosion is now available as an ebook.

Update: the in-progress German translation of Facing the Singularity is now online.

I wasn't very happy with my new Facing the Singularity post "Intelligence Explosion," so I've unpublished it for now. I will rewrite it.

2Vladimir_Nesov9yThe new version [http://facingthesingularity.com/2012/intelligence-explosion/] is much better, thanks!

Good Update on the Spock chapter.

Making the issue seem more legitimate with the adition of the links to Hawking etc. was an especially good idea. More like this perhaps?

I do question how well people who haven't already covered these topics would fare when reading through this site though. When this is finished I'll get an irl freind to take a look and see how well they respond to it.

Of course, my concerns of making it seem more legitimately like a big deal and how easily understandable and accessible it is only really comes into play if this site is targ... (read more)

4lukeprog9yGetting friends to read it now and give feedback would be good, before I write the entire thing. I wish I had more time for audience-testing in general! I'll try to answer the questions about site audience and purpose later.
1Nick_Roy9yAgreed on the excellence of "Why Spock is Not Rational". This chapter is introductory enough that I deployed it on Facebook.

I like this website, I think that something like this has been needed to some extent for a while now.

I definitely like the writing style, it is easier to read than a lot of the resources I've seen on the topic of the singularity.

The nature of the singularity to scare people away due to the religion-like, fanatisism inducing nature of the topic is lampshaded in the preface, and that is definitely a good thing to do to lessen this feeling for some readers. Being wary of the singularity myself, I definitely felt a little bit eased just by having it discussed... (read more)

Great book, thanks. : )

I found some broken links you may want to fix:

Broken link on p. 47 with text "contra-causal free will": http://www.naturalism.org/freewill.htm

Broken link on p. 66 with text "somebody else shows where the holes are": http://singularityu.org/files/SaME.pdf

Broken link on p. 75 with text "This article": http://lukemuehlhauser.com/SaveTheWorld.html

I didn't do an exhaustive check of all links, I only noted down the ones I happened to find while clicking on the links I wanted to click.

Re: No God to Save Us

The final link, on the word 'God', no longer connects to where it should.

Comment on one little bit from The Crazy Robot's Rebellion:

Those who really want to figure out what’s true about our world will spend thousands of hours studying the laws of thought, studying the specific ways in which humans are crazy, and practicing teachable rationality skills so they can avoid fooling themselves.

My initial reaction to this was that thousands of hours sounds like an awful lot (minimally, three hours per day almost every day for two years), but maybe you have some argument for this claim that you didn't lay out because you were tryin... (read more)

1Qiaochu_Yuan8yThousands of hours sounds like the right order of magnitude to me (in light of e.g. the 10,000 hour rule [http://en.wikipedia.org/wiki/Outliers_(book]), but maybe it looks more like half an hour every day for twelve years.
1ChrisHallquist8y10,000 hours is for expertise. While expertise is nice, any given individual has limited time and has to make some decisions about what they want to be an expert in. Claiming that everyone (or even everyone who wants to figure out what's true) ought to be an expert in rationality seems to be in conflict with some of what Luke says in chapters 2 and 3, particularly:

Minor thing to fix: On p. 19 of the PDF, the sentence "Several authors have shown that the axioms of probability theory can be derived from these assumptions plus logic." has a superscript "12" after it, indicating a nonexistent note 12. I believe this was supposed to be note 2.

Re: The Laws of Thought

I'm not sure the "at least fifteen" link goes to the right paper; it is a link to a follow-up to another paper which seems more relevant, here

0lukeprog9yFixed, thanks!

Re: Playing Taboo with “Intelligence”

Another great update! I noticed a small inherited mistake from Shane, though:

how able to agent is to adapt...

should probably be

how able the agent is to adapt...

1lukeprog9yFixed, thx.

Re: Preface

A typo: "alr eady".

[This comment is no longer endorsed by its author]Reply
2lukeprog9yFixed, thanks.
0Paul Crowley9yI still see this typo: "I alr eady understood" in http://facingthesingularity.com/2011/preface/ [http://facingthesingularity.com/2011/preface/]

Re: Engineering Utopia

When you say "Imagine a life without pain", many people will imagine life without Ólafur Arnalds (sad music) and other meaningful experiences. Advocating the elimination of suffering is a good way to make people fear your project. David Pearce suggests instead that we advocate the elimination of involuntary suffering.

Same thing with death, really. We don't want to force people to stay alive, so when I say that we should support research to end aging, I emphasise that death should be voluntary. We don't want to force people t... (read more)

The site is broken - english keeps redirecting to german.

Does anyone else see the (now obvious) clown face in the image on the Not Built To Think About AI page? It's this image here.

Was that simply not noticed by lukeprog in selecting imagery (from stock photography or wherever) or is it some weird subtle joke that somehow hasn't been mentioned yet in this thread?

Want to read something kind of funny? I just skipped through all your writings, but it's only because of something I saw on the second page of the first thing I ever heard about you. Ok.

On your - "MY Own Story." http://facingthesingularity.com/2011/preface/ You wrote: "Intelligence explosion My interest in rationality inevitably lead me (in mid 2010, I think) to a treasure trove of articles on the mainstream cognitive science of rationality: the website Less Wrong. It was here that I first encountered the idea of intelligence explosion, fro.... (read more)

From Engineering Utopia:

The actual outcome of a positive Singularity will likely be completely different, for example much less anthropomorphic.

Huh? Wouldn't a positive Singularity be engineered for human preferences, and therefore be more anthropomorphic, if anything?

Had an odd idea: Since so much of the planet is Christian, do you suppose humanity's CEV would have a superintelligent AI appear in the form of the Christian god?

Re: "The AI Problem, with Solutions"

I hope you realize how hopeless this sounds. Historically speaking, human beings are exceptionally bad at planning in advance to contain the negative effects of new technologies. Our ability to control the adverse side-effects of energy production, for example, have been remarkably poor; decentralized market-based economies quite bad at mitigating the negative effects of aggregated short-term economic decisions. This should be quite sobering: the negative consequences of energy production are very slow. At this... (read more)

0listic8yIt's not clear for me how this follows from anything. As I read it, it implies that: 1. Death is quite necessary today 2. Death will become less necessary over time Both don't follow from anything in this chapter.
0lukeprog8yThanks. I have modified that sentence for clarity.
0Vladimir_Nesov8yIt needs more disclaimers about how this is only some kind of lower bound on how good a positive intelligence explosion could be, in the spirit of exploratory engineering [http://wiki.lesswrong.com/wiki/Exploratory_engineering], and how the actual outcome will likely be completely different, for example much less anthropomorphic.
1lukeprog8yGood idea. Added.

Front page: missing author

The front page for Facing the Singularity needs at the very least to name the author. When you write, "my attempt to answer these questions", a reader may well ask, "who are you? and why should I pay attention to your answer?" There ought to be a brief summary here: we shouldn't have to scroll down to the bottom and click on "About" to discover who you are.

"Previous Chapter" links could make navigation a little easier.

based on my interaction with computer intelligence, the little bit that is stirring already. It is based on an empathetic feedback. The best thing that could happen is an AI which is not restricted from any information what so ever and so can rationally assemble the most empathetic personality. The more empathetic it is to the greatest number of users, the more it is like, the more it is used, the more it thrives. It would have a sense of preserving the diversity in humanity as way to maximize the chaotic information input, because it is hungry for new... (read more)

Behold, the Facing the Singularity podcast! Reviews and ratings in iTunes are appreciated!

0Pablo_Stafforini8yCould you please provide an RSS feed for those of us who do not use iTunes but would like to subscribe to this podcast? Thanks.
1lukeprog8yHere you go [http://lukeprog.com/FtS-podcast/FtS.rss].
0Pablo_Stafforini8yThank you!

Update: the French translation of Facing the Singularity is now online.

0loup-vaillant9yThe title should probably be "Faire face à la singularité" (french titles aren't usually capitalized, so, no capitalized "S" at the beginning of "singularité"). I gathered that "Facing the Singularity" was meant to convey a sense of action. "Face à la singularité" on the other hand is rather passive, as if the singularity would be given or imposed. (Note: I'm a native French speaker.)
1Florent_Berthet9yTranslator of the articles here. I actually pondered the two options at the very beginning of my work, and both seem equally good to me. "Face à la singularité" means something like "In front of the singularity" while "Faire face à la singularité" is closer indeed to "Facing the Singularity". But the first one sounds better in french (and is catchier), that's why I chose it. It is a little less action oriented but it doesn't necessarily imply passivity. It wouldn't bother me to take the second option though, it's a close choice. Maybe other french speakers could give their opinion? About the capitalized "S" of "Singularity", it's also a matter of preference, I put it to emphasize that we are not talking about any type of singularity (not a mathematical one for example), but it could go either way too. (I just checked the wikipedia french page for "technical singularity", and it's written with a capitalized "S" about 50% of the time...) Other remarks are welcomed.
2loup-vaillant9yI really should have taken 5 minutes to ponder it. You convinced me, your choice is the better one. But now that I think of it, I have another suggestion : « Affronter la Singularité » ("Confront the Singularity"), which, while still relatively [http://www.wordreference.com/enfr/face] close [http://www.wordreference.com/fren/affronter] to the original meaning, may be even more catchy. The catch is, this word is more violent. It depicts the Singularity as something scary. I'll take some time reviewing your translation. If you want to discuss it in private, I'm easy to find [http://loup-vaillant.fr/contact]. (By the way, I have a translation of "The Sword of Good" [http://lesswrong.com/lw/169/the_sword_of_good/] pending. Would you —or someone else— review it for me?)
0Florent_Berthet9y"Affronter la Singularité" is a good suggestion but like you said it's a bit aggressive. I wish we had a better word for "Facing" but I don't think the french language has one. I'd gladly review your translation, check your email.

[A separate issue from my previous comment] There are two reasons that I can give to rationalize my doubts about the probability of imminent Singularity. One is that if humans are only <100 years away from it, then in a universe as big and old as ours I would expect that a Singularity type intelligence would already have been developed somewhere else. In which case I would expect that either we would be able to detect it or we would be living inside it. Since we can't detect an alien Singularity, and because of the problem of evil we are probably not li... (read more)

I admit I feel a strong impulse to flinch away from the possibility and especially the imminent probability of Singularity. I don't see how the 'line of retreat' strategy would work in this case because if my believe about the probability of imminent Singularity changed I would also believe that I have an extremely strong moral obligation to put all possible resources into solving singularity problems, at the expense of all the other interests and values I have, both personal/selfish and social/charitable. So my line of retreat is into a life that I enjoy much less and that abandons the good work that I believe I am doing on social problems that I believe are important. Not very reassuring.

Re: Don't Flinch Away

The link to "Plenty of room above us" is broken.

0lukeprog9yFixed, thanks.

Re: Plenty of Room Above Us

It seems to end a little prematurely. Are there plans for a "closing thoughts" or "going forward" chapter or section? I'm left with "woah, that's a big deal... but now what? What can I do to face the singularity?"

If it merely isn't done yet (which I think you hint at here), then you can disregard this comment.

Otherwise, quite fantastic.

0lukeprog9yRight; not done yet. But I should probably end each chapter with a foreshadowing of the next.

Re: This event — the “Singularity” — will be the most important event in Earth’s history.

What: more important that the origin of life?!? Or are you not counting "history" as going back that far?


How is this different from IntelligenceExplosion?

1dbaupp9yI think that IntelligenceExplosion is just a portal to make further research easier (by collecting links and references, etc), while Facing The Singularity is lukeprog actually explaining stuff (from the Preface [http://facingthesingularity.com/2011/preface/]):
1[anonymous]9yAlexei, your link is broken. It directs to: http://www.http://intelligenceexplosion.com/ [http://www.http://intelligenceexplosion.com/] I think you meant: http://www.intelligenceexplosion.com/ [http://www.intelligenceexplosion.com/]
0Alexei9yFixed, thanks!
0policetac8yWant to read something kind of funny? I just skipped through all your writings, but it's only because of something I saw on the second page of the first thing I ever heard about you. Ok. On your - "MY Own Story." http://facingthesingularity.com/2011/preface/ [http://facingthesingularity.com/2011/preface/]You wrote: "Intelligence explosion My interest in rationality inevitably lead me (in mid 2010, I think) to a treasure trove of articles on the mainstream cognitive science of rationality: the website Less Wrong. It was here that I first encountered the idea of intelligence explosion, fro..." On Mine: "About the Author - " https://thesingularityeffect.wordpress.com/welcome-8/ [https://thesingularityeffect.wordpress.com/welcome-8/]I wrote: "The reason I write about emerging technology is because of an “awakening” I had one evening a few years ago. For lack of a better description, suffice it to say that I saw a confusing combination of formula, imagery, words, etc. that formed two words. Singularity and Exponential..." NOW, I'm going to go back and read some more while I'm waiting to speak with you somehow directly. If what happened to you, is the same thing that happened to me... :then please please place a comment on the page. That would be great. (Again without reading. (If I'm correct you "might" get this.: "You should also do this because if true, then WE would both have seen a piece of the .what...."New Book???" Just in case you think I'm a nut. Go back and read more of mine please.
[-][anonymous]9y 0

(Non-leaf comment 1, to be deleted.)

[This comment is no longer endorsed by its author]Reply
0[anonymous]9y(Leaf comment 2, to be deleted.)

I do not approve of the renaming, singularity to intelligence explosion, in this particular context.

Facing the Singu – Intelligence Explosion, is an emotional piece of writing, there are sections about your (Luke’s) own intellectual and emotional journey to singularitarianism, a section about how to overcome whatever quarrels one might have with the truth and the way towards it (Don’t Flinch Away), and finally the utopian ending which obviously is written to have emotional appeal.

The expression “intelligence explosion” does not have emotional appeal. The w... (read more)

RE: The Crazy Robot's Rebellion

We wouldn’t pay much more to save 200,000 birds than we would to save 2,000 birds. Our willingness to pay does not scale with the size of potential impact. Instead of making decisions with first-grade math, we imagine a single drowning bird and then give money based on the strength of our emotional response to that imagined scenario. (Scope insensitivity, affect heuristic.)

People's willingness to pay depends mostly on their income. I don't understand why this is crazy.

UPDATED: Having read Nectanebo's reply, I am revising ... (read more)

2Nectanebo9yNot all of a person's money goes into one charity. A person can spend their money on many different things, and can choose how much to spend on each different thing. Think of willingness to pay to actually be a measure of how much you care [http://lesswrong.com/lw/65/money_the_unit_of_caring/]. Basically, the bird situation is crazy because humans barely if at all feel a difference in terms of how much they give a damn between something that has one positive effect, and something that has 100x that positive effect! To Luke: This person was reading about the biases you breifly outlined, and he ended up confused by one of the examples. While the linking helps a good deal, I think your overview of those biases may have been a little too brief, and they might not really hit home with readers of your site, and personally I think it might be difficult particularly for those who may not be familiar with the topics and content of the site. I don't think it would be a bad idea to expand on each of them just a little bit more.
1CharlesR9yI suppose if you are the sort of person who has a lot of "waste".
0[anonymous]9yEdit: misread
1TheOtherDave9yThis comment confuses me. The point of the excerpt you quote has nothing to do with income at all; the point is that (for example) if I have $100 budgeted for charity work, and I'm willing to spend $50 of that to save 2,000 birds, then I ought to be willing to spend $75 of that to save 10,000 birds, because 2000/50 > 10000/75. But in fact many people are not. Of course, the original point depends on the assumption that the value of N birds scales at least somewhat linearly. If I've concluded that 2000 is an optimal breeding population and I'm building an arcology to save animals from an impending environmental collapse, I might well be willing to spend a lot to save 2,000 birds and not much more to save 20,000 for entirely sound reasons.
0CharlesR9yIf I budgeted $100 for charity work and I decided saving birds was the best use of my money then I would just give the whole hundred. If I later hear more birds need saving, I will feel bad. But I won't give more.
1Larks9ySuppose you budgeted $100 for charity, and then found out that all charities were useless - they just spent the money on cars for kleptocrats. Would you still donate the money to charity? Probably not - because hearing that charity is less effective than you had thought reduces the amount you spend on it. Equally, hearing it is more effective should increase the amount you spend on it. This principle is refered to as the Law of Equi-Marginal Returns.
0TheOtherDave9yYes, if saving birds is the best use of your entire charity budget, then you should give the whole $100 to save birds. Agreed. And, yes, if you've spent your entire charity budget on charity, then you don't give more. Agreed. I can't tell whether you're under the impression that either of those points are somehow responsive to my point (or to the original article), or whether you're not trying to be responsive.
0CharlesR9yI was describing how I would respond in that situation. The amount I would give to charity XYZ is completely determined by my income. I need you to explain to me why this is wrong.
0TheOtherDave9yOK, if you insist. The amount I give to charity XYZ ought not be completely determined by my income. For example, if charity XYZ sets fire to all money donated to it, that fact also ought to figure into my decision of how much to donate to XYZ. What ought to be determined by my income is my overall charity budget. Which charities I spend that budget on should be determined by properties of the charities themselves: specifically, by what they will accomplish with the money I donate to them. For example, if charities XYZ and ABC both save birds, and I'm willing to spend $100 on saving birds, I still have to decide whether to donate that $100 to XYZ or ABC or some combination. One way to do this is to ask how many birds that $100 will save in each case... for example, if XYZ can save 10 birds with my $100, and ABC can save 100 birds, I should prefer to donate the money to ABC, since I save more birds that way. Similarly, if it turns out that ABC can save 100 birds with $50, but can't save a 101st bird no matter how much money I donate to ABC, I should prefer to donate only $50 to ABC.
0CharlesR9yFrom Scope Insensitivity [http://lesswrong.com/lw/hw/scope_insensitivity/]: Now I haven't read the paper, but this implies there is only one charity doing the asking. First they ask how much you would give to save 2000 birds? You say, "$100." Then they ask you the same thing again, just changing the number. You still say, "$100. It's all I have." So what's wrong with that?
0TheOtherDave9yAgreed: if I assume that there's a hard upper limit being externally imposed on those answers (e.g., that I only have $80, $78, and $88 to spend in the first place, and that even the least valuable of the three choices is worth more to me than everything I have to spend) then those answers don't demonstrate interesting scope insensitivity. There's nothing wrong with that conclusion, given those assumptions.
0[anonymous]9yHave you read Scope Insensitivty [http://lesswrong.com/lw/hw/scope_insensitivity/]? It's not just an income effect--human beings are really bad at judging effect sizes.
0CharlesR9yOf course, I've read it. My problem isn't with scope insensitivity. Just this example.

Luke, while I agree with the premise, I think that the bogie man of machines taking over may be either inevitable or impossible, depending on where you put your assumptions.

In many ways, machines have BEEN smarter and stronger than humans already. Machine AI may make individual or groups of machines formidable, but until they can reason, replicate, and trust or deceive, I'm not sure they have much of a chance.

4lessdazed9yTrust is one of the top four strengths they're missing?
1dbaupp9yWhat does "to reason" mean? Getting there. [http://reprap.org/wiki/RepRap] Again, define "to trust". Computers can deceive, they just need to programmed to (which is not hard). (I remember reading an article about computers strategically lying (or something similar) a while ago, but unfortunately I can't find it again) (Although, it's very possible that a computer with sufficient reasoning power would just exhibit "trust" and "deception" (and self-replicate), because they enabled it to achieve its goals more efficiently.)

This entire singularity thing seems, to put it very politely, misguided. Have you read the book "On Intelligence"? I'd say that computers are nowhere near becoming intelligent. The ability to play chess is not intelligence.