Open Thread: February 2010, part 2

by CronoDAS1 min read16th Feb 2010917 comments


Open Threads
Personal Blog

The Open Thread posted at the beginning of the month has gotten really, really big, so I've gone ahead and made another one. Post your new discussions here!

This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.

Rendering 500/917 comments, sorted by (show more) Highlighting new comments since Today at 4:02 AM
New Comment
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings
[-][anonymous]11y 36

So, I walked into my room, and within two seconds, I saw my laptop's desktop background change. I had the laptop set to change backgrounds every 30 minutes, so I did some calculation, and then thought, "Huh, I just consciously experienced a 1-in-1000 event."

Then the background changed again, and I realized I was looking at a screen saver that changed every five seconds.

Moral of the story: 1 in 1000 is rare enough that even if you see it, you shouldn't believe it without further investigation.

That is a truly beautiful story. I wonder how many places there are on Earth where people would appreciate this story.

No! Not for a second! I immediately began to think how this could have happened. And I realized that the clock was old and was always breaking. That the clock probably stopped some time before and the nurse coming in to the room to record the time of death would have looked at the clock and jotted down the time from that. I never made any supernatural connection, not even for a second. I just wanted to figure out how it happened.

-- Richard P Feynman, on being asked if he thought that the fact that his wife's favorite clock had stopped the moment she died was a supernatural occurrence, quoted from Al Sekel, "The Supernatural Clock"

2RichardKennaway11yThis should be copied to the Rationality Quotes thread.
4Paul Crowley11yThere are a lot of opportunities in the day for something to happen that might prompt you to think "wow, that's one in a thousand", though. It wouldn't have been worth wasting a moment wondering if it was coincidence unless you had some reason to suspect an alternative hypothesis, like that it changed because the mouse moved. bit that makes no sense deleted
2lunchbox11yExactly. []
1Document9yRecently posted to Reddit. [] (Edit: About three days later I realized that that's a 1/100 and not a 1/1000 chance; my bad.)

I've been finding PJ Eby's article The Multiple Self quite useful for fighting procrastination and needless feelings of guilt about getting enough done / not being good enough at things.

I have difficulty describing the article briefly, as I'm afraid that I accidentally omit important points and make people take it less seriously than it deserves, but I'll try. The basic idea is that the conscious part of our mind only does an exceedingly small part of all the things we spend doing in our daily lives. Instead, it tells the unconscious mind, which actually does everything of importance, what it should be doing. As an example - I'm writing this post right now, but I don't actually consciously think about hitting each individual key and their exact locations on my keyboard. Instead I just tell my mind what I want to write, and "outsource" the task of actually hitting the keys to an "external" agent. (Make a function call to a library implementing the I/O, if you want to use a programming metaphor.) Of course, ultimately the words I'm writing come from beyond my conscious mind as well. My conscious mind is primarily concerned with communicating Eby's point well to my... (read more)

5xamdam11yThis somehow reminds me of the stories when Tom Schelling was trying to quit smoking, using game theory against himself (or his other self). The other self in question was not the unconscious, but the conscious "decision-making" self in different circumstances. So that discussion is somewhat orthogonal to this one. I think he did things like promising to give a donation to the American Nazi Party if he smokes. Not sure how that round ended, but he did finally quit.
6Jack11yHmm. I'd be worried it'd backfire and I'd start subtlety disliking Jews. Then you're a smoker and a bigot.
2xamdam11ylol. Not a problem if you're Jewish ;)
3Jack11ySelf-hatred is even worse than being a bigot!
2khafra11yReminds me of The User Illusion [] , which adds that the consciousness has an astoundingly low bandwidth--around 16bps--around 6 orders of magnitude lower than the senses transmit to the brain.
2CronoDAS11yInteresting. I've glanced at that site before and its metaphors have the ring of truthiness (in a non-pejorative sense) about them; the programming metaphors and the focus on subconscious mechanisms seem to resonate with the way I already think about how my own brain works.
3RobinZ11yCouldn't that be more succinctly stated as "its metaphors have the ring of truth about them"?
3CronoDAS11yMaybe, but a lot of Freud's metaphors had/have a similar ring.
1xamdam11yI read the original article and some of the other PJE material. I think he's really onto something. This is how far I got: * Identify the '10% controlling part' * Everything else is not under direct control (which is where most self-help methods fail) * It is under indirect control So far makes sense from personal experience/general knowledge. * Here are my methods for indirect control. This is the part that I remain skeptical about . Not PJE's fault, but I do need more data/experience to confirm.

Just a general comment about this site: it seems to be biased in favor of human values at the expense of values held by other sentient beings. It's all about "how can we make sure an FAI shares our [i.e. human] values?" How do you know human values are better? Or from the other direction: if you say, "because I'm human", then why don't you talk about doing things to favor e.g. "white people's values"?

I wish the site were more inclusive of other value systems ...

8mattnewport11yThis site does tend to implicitly favour a subset of human values, specifically what might be described as 'enlightenment values'. I'm quite happy to come out and explicitly state that we should do things that favour my values, which are largely western/enlightenment values, over other conflicting human values.
6Clippy11yAnd I think we should pursue values that aren't so apey. Now what?
4mattnewport11yYou're outnumbered.
2hal900011yOnly by apes. And not for long. If we're voting on it, the only question is whether to use viral values or bacterial values.
7Alicorn11yToo long has the bacteriophage menace oppressed its prokaryotic brethren! It's time for an algaeocracy!
3mattnewport11yTrue, outnumbered was the wrong word. Outgunned might have been a better choice.
4[anonymous]11yWhite people value the values of non-white people. We know that non-white people exist, and we care about them. That's why the United States is not constantly fighting to disenfranchise non-whites. If you do it right, white people's values are identical to humans' values.
9Clippy11yHi there. It looks like you're speaking out of ignorance regarding the historical treatment of non-whites by whites. Please choose the country you're from: United Kingdom [] United States [] Australia [] Canada [] South Africa [] Germ... nah, you can figure that one out for yourself.
4cousin_it11yWhat other sentient beings? As far as I know, there aren't any. If we learn about them, we'll probably incorporate their well-being into our value system.
5Clippy11yYou mean like you advocated doing to the "Baby-eaters"? (Technically, "pre-sexual-maturity-eaters", but whatever.) ETA: And how could I forget this []?
4Nick_Tarleton11yI have no idea if this is a serious question, but.... "Better"? See Invisible Frameworks []. We don't say that. See No License To Be Human [].
3Rain11yI approve of Clippy providing a roleplay exercise for the readers, and am disappointed in those who treat it as a "joke" when the topic is quite serious. This is one of my two main problems with ethical systems in general: 1) How do you judge what you should (value-judgmentally) value? 2) How do you deal with uncertainty about the future (unpredictable chains of causality)? Eliezer's "morality" and "should" definitions do not solve either of these questions, in my view.
3wedrifid11yWe more or less do. Or rather we favour values of a distinct subset humanity and not the whole.
5Nick_Tarleton11yWe don't favor those values because they are the values of that subset — which is what "doing things to favor white people's values" would mean — but because we think they're right. (No License To Be Human [], on a smaller scale.) This is a huge difference.
3wedrifid11yGiven the way I use 'right' this is very nearly tautological. Doing things that favour my values is right by (parallel) definition.
2Roko11ySure, we favor the particular Should Function that is, today, instantiated in the brains of roughly middle-of-the-range-politically intelligent westerners.
6Clippy11yWell, you shouldn't.
2Vladimir_Nesov11yDo you think there is no simple procedure that would find roughly the same "should function" hidden somewhere in the brain of a brain-washed blood-thirsty religious zealot? It doesn't need to be what the person believes, what the person would recognize as valuable, etc., just something extractable from the person, according to a criterion that might be very alien to their conscious mind. Not all opinions (beliefs/likes) are equal, and I wouldn't want to get stuck with wrong optimization-criterion just because I happened to be born in the wrong place and didn't (yet!) get the chance to learn more about the world. (I'm avoiding the term 'preference' to remove connotations I expect it to have for you, for what I consider the wrong reasons.)
2Roko11yA lot of people seem to want to have their cake and eat it with CEV. Haidt has shown us that human morality is universal in form and local in content, and has gone on to do case studies showing that there are 5 basic human moral dimensions (harm/care, justice/fairness, loyalty/ingroup, respect/authority, purity/sacredness), and our culture only has the first two. It seems that there is no way you can run an honestly moraly neutral CEV of all of humanity and expect to reliably get something you want. You can either rig CEV so that it tweaks people who don't share our moral drives, or you can just cross your fingers and hope that the process of extrapolation causes convergence to our idealized preferences, and if you're wrong you'll find yourself in a future that is suboptimal.
1CarlShulman11yHaidt just claims that the relative balance of those five clusters differ across cultures, they're present in all.
1Vladimir_Nesov11yOn one hand, using preference-aggregation is supposed to give you the outcome preferred by you to a lesser extent than if you just started from yourself. On the other hand, CEV is not "morally neutral". (Or at least, the extent to which preference is given in CEV implicitly has nothing to do with preference-aggregation.) We have a tradeoff between the number of people to include in preference-aggregation and value-to-you of the outcome. So, this is a situation to use the reversal test []. If you consider only including the smart sane westerners as preferable to including all presently alive folks, then you need to have a good argument why you won't want to exclude some of the smart sane westerners as well, up to a point of only leaving yourself.
1Roko11yYes, a CEV of only yourself is, by definition optimal. The reason I don't recommend you try it is because it is infeasible; probability of success is very low, and by including a bunch of people who (you have good reason to think) are a lot like you, you will eventually reach the optimal point in the tradeoff between quality of outcome and probability of success.
2Unknowns11yI hope you realize that you are in flat disagreement with Eliezer about this. He explicitly affirmed that running CEV on himself alone, if he had the chance to do it, would be wrong.
1wedrifid11yEliezer quite possibly does believe that. That he can make that claim with some credibility is one of the reasons I am less inclined to use my resources to thwart Eliezer's plans for future light cone domination. Nevertheless, Roko is right more or less by definition and I lend my own flat disagreement to his.
1Eliezer Yudkowsky11yConfirmed.
1Vladimir_Nesov11y"Low probability of success" should of course include game-theoretic considerations where people are more willing to help you if you give more weight to their preference (and should refuse to help you if you give them too little, even if it's much more than status quo, as in Ultimatum game). As a rule, in Ultimatum game you should give away more if you lose from giving it away less. When you lose value to other people in exchange to their help, having compatible preferences doesn't necessarily significantly alleviate this loss.
1Roko11ySure, the kolmogorov complexity of a set of edits to change the moral reflective equilibrium of a human is probably pretty low compared to the complexity of the overall human preference set. But that works the other way around too. Somewhere hidden in the brain of a a liberal western person is a murderer/terrorist/child abuser/fundamentalist if you just perform the right set of edits.
2Vladimir_Nesov11yAgain, not all beliefs are equal. You don't want to use the procedure that'll find a murderer in yourself, you want to use the procedure that'll find a nice fellow in a murderer. And given such a procedure, you won't need to exclude murderers from extrapolated volition.
2Nick_Tarleton11yYou seem uncharacteristically un-skeptical of convergence within that very large group, and between that group and yourself.
2LucasSloan11yI'm pretty sure that I'm not against simply favoring the values of white people. I expect that a CEV performed on only people of European descent would be more or less indistinguishable from that of humanity as a whole.
2Kutta11yDepending on your stance about the psychological unity of mankind you could even say that the CEV of any sufficiently large number of people would greatly resemble the CEV of other posible groups. I personally think that even the CEV of a bunch of Islamic fundamentalists would suffice for enlightened western people well enough.

Here's something interesting on gender relations in ancient Greece and Rome.

Why did ancient Greek writers think women were like children? Because they married children - the average woman had her first marriage between the ages of twelve and fifteen, and her husband would usually be in his thirties.

2bgrah44911yThe reason ancient Greek writers thought women were like children is the same reason men in all cultures think women are like children: There are significant incentives to do so. Men who treat women as children reap very large rewards compared to those men who treat women as equals. EDIT: If someone thinks this is an invalid point, please explain in a reply. If the downvote(s) is just "I really dislike anyone believing what he's saying is true, even if a lot of evidence supports it" (regardless of whether or not evidence currently supports it) then please leave a comment stating that. EDIT 2: Supporting evidence or retraction will be posted tonight. EDIT 3: As I can find no peer-reviewed articles suggesting this phenomenon, I retract this statement.

This conversation has been hacked.

The parent comment points to an article presenting a hypothesis. The reply flatly drops an assertion which will predictably derail conversation away from any discussion of the article.

If you're going to make a comment like that, and if you prefix it with something along the lines of "The hypothesis in the article seems superfluous to me; men in all cultures treat women like children because...", and you point to sources for this claim, then I would confidently predict no downvotes will result.

(ETA: well, in this case the downvote is mine, which makes prediction a little too easy - but the point stands.)

6Roko11yLW doesn't like to hear the truth about male/female sexual strategies; we like to have accurate maps here, but there's a big "censored" sign over the bit of the map that describes the evolutionary psychology of sexuality, practical dating advice, the burgeoning "pick-up" community and an assorted cloud of topics. Reasons for this censorship (and I agree to an extent) are that talking about these topics offends people and splits the community. LW is more useful, it has been argued, if we just don't talk about them.
8Morendil11yThe PUA community include people who come across as huge assholes, and that could be an alternative explanation of why people react negatively to the topics, by association. I'm thinking in particular of the blog "Roissy in DC", which is on OB's blogroll. Offhand, it seems to me that thinking of all women as children entails thinking of some adults as children, which would be a map-territory mistake around the very important topic of personhood. I did pick up some interesting tips from PUA writing, and I do think there can be valuable insight there if you can ignore the smell long enough to dig around (and wash your hands afterwards, epistemically speaking). No relevant topics should be off-limits to a community of sincere inquiry. Relevance is the major reason why I wouldn't discuss the beauty of Ruby metaprogramming on LessWrong, and wouldn't discuss cryonics on a project management mailing list. If discussions around topic X systematically tend to go off the rails, and topic X still appears relevant, then the conclusion is that the topic of "why does X cause us to go off the rails" should be adequately dealt with first, in lexical priority. That isn't censorship, it's dependency management.
5Roko11yBut in reality, this topic is off-limits. Therefore LW is not a community of sincere inquiry, but nothing's perfect, and LW does a lot of good. Interesting. However, in this case, that discussion might get somewhat accusatory, and go off the rails itself.
5Morendil11yGot that. I am suggesting that it is off-limits because this community isn't yet strong enough at the skills of collaborative truth-seeking. Past failures shouldn't be seen as eternal limitations; as the community grows, by acquiring new members, it may grow out of these failures. To make this concrete, the community seems to have a (relative) blind spot around things like pragmatics, as well as what I've called "myths of pure reason". One of the areas of improvement is in reasoning about feelings. I'm rather hopeful, given past contributions by (for instance) Alicorn and pjeby.
3Roko11yI don't think that is the reason for the problem. The community doesn't go off the rails and have to censor discussions about merely pragmatic issues. It is more that the community has a bias surrounding the concept of traditional, storybook-esque morality, roughly a notion of doing good that seems to have some moral realist heritage, a heavy tint of political correctness, and sees the world in black-and-white terms, rather than moral shades of grey. Facts that undermine this conception of goodness can't be countenanced, it seems. Robin Hanson, on the other hand, has no trouble posting about the sexuality/seduction cluster of topics. There seems to be a systematic difference between OB and LW along this "moral political correctness/moral constraints" dimension - Robin talks with enthusiasm about futures where humans have been replaced with Vile Offspring, and generally shuns any kind of talk about ethics. (EDITED, thanks to Morendil)
4wedrifid11yThere is truth to what you say but unfortunately you are letting your frustration become visible. That gives people the excuse to assign you lower status and freely ignore your insight. This does not get you what you want. This is perhaps one of the most important lessons to be learned on the topic of 'pragmatics'. Whether you approach the topic from works like Robert Greene []'s on Power [], War [] and Seduction [] or from the popular social skills based self help communities previously mentioned [], a universal lesson is that things aren't fair, bullshit is inevitable and getting indignant about the bullshit gets in the way of your pragmatic goals. There may be aspects of the morality here that is childlike or naive and I would be interested in your analysis of the subject since you clearly have given it some thought. But if you are reckless and throw out 'like theists' references without thought your contribution will get downvoted to oblivion and I will not get to hear what you have to say. Around here that more or less invokes the 'nazi' rule. Edit: No longer relevant.
2Roko11yLOL... indeed. I am not sure that I am actually, in far mode, so interested in correcting this particular LW bias. In near mode, SOMEONE IS WRONG ON THE INTERNET bias kicks in. It seems like it'll be an uphill struggle that neither I nor existential risk mitigation will benefit from. A morally naive LW is actually good for X-risks, because that particular mistake (the mistake of thinking in terms of black-and-white morality and Good and Evil) will probably make people more "in the mood" for selfless acts of charity.
3wedrifid11yI think I agree. If Eliezer didn't have us all convinced that he is naive in that sense we would probably have to kill him before he casts his spell of ultimate power. (cough The AI Box demonstrations were just warm ups...)
4Morendil11yThis kind of phrase seems designed to rile (some of) your readers. You will improve the quality of discourse substantially by understanding that and correcting for it. Unless, of course, your goal really is to rile readers rather than to improve quality of discourse.
3wedrifid11yRobin can do what he likes on his own blog without direct consequences within the blog environment. He also filters which comments he allows to be posted. I guess what I am saying is that it isn't useful to compare OB and LW on this dimension because the community vs individual distinction is far more important than the topic clustering.
2Morendil11yI may not have been clear: I meant pragmatics in this sense [], roughly "how we do things with words". I'd also include things like denotation vs connotation in that category. Your comment on "pragmatic issues" suggests you may have understood another sense.
3gwern11yThe article suggests a direct counter-example: by having high standards, the men forfeit the labor of the women in things like 'help[ing] with finance and political advice'. Much like the standard libertarian argument against discrimination: racists narrow their preferences, raising the cost of labor, and putting themselves at a competitive disadvantage. Men may as a group have incentive to keep women down, but this is a prisoner's dilemma.
3Paul Crowley11yWhy do so many people here believe that? It strongly contradicts my experience.
8cousin_it11yYour experience is atypical because you're atypical [].
2Paul Crowley11yMan, I've barely looked at that page since I wrote it four years ago. I live with Jess now, across the road from the other two. I can heartily recommend my brand of atypicality :-)
2thomblake11yGood answer. I keep privately asking the same question about these sorts of things, and getting the same answer from others.
3Vladimir_Nesov11yWhat do you mean by "many people here believe that"? Believe what? And what tells you they do believe it?
2Douglas_Knight11yPeople have great difficulty verbalizing their perceptions of and beliefs about social interactions. It is not obvious to me that you two have different beliefs or experiences. More likely, you do, but it would probably take a lot of work to identify and communicate those differences.

Are people interested in reading an small article about a case of abuse of frequentist statistics? (In the end, the article was rejected, so the peer review process worked.) Vote this comment up if so, down if not. Karma balance below.

ETA: Here's the article.

1Douglas_Knight11yIf it's really frequentism that caused the problem, please spell this out. I find that "frequentist" is used a lot around here to mean "not correct." (but I'm interested whether or not it's about frequentism)
2Technologos11yMy understanding is that one primary issue with frequentism is that it can be so easily abused/manipulated to support preferred conclusions, and I suspect that's the subject of the article. Frequentism may not have "caused the problem," per se, but perhaps it enabled it?

Until yesterday, a good friend of mine was under the impression that the sun was going to explode in "a couple thousand years." At first I thought that this was an assumption that she'd never really thought about seriously, but apparently she had indeed thought about it occasionally. She was sad for her distant progeny, doomed to a fiery death.

She was moderately relieved to find out that humanity had millions of times longer than she had previously believed.

8sketerpot11yI wonder how many trivially wrong beliefs we carry around because we've just never checked them. (Probably most of them are mispronunciations of words [], at least for people who've read a lot of words they've never heard anybody else use aloud.) For the longest time, I thought that nuclear waste was a green liquid that tended to ooze out of barrels. I was surprised to learn that it usually came in the form of dull gray metal rods.
2wedrifid11yDoes it still give you superpowers?

If you extract the plutonium and make enough warheads, and you have missiles capable of delivering them, it can make you a superpower in a different sense. I'm assuming that you're a large country, of course.

More seriously, nuclear waste is just a combination of the following:

  1. Mostly Uranium-238, which can be used in breeder reactors.

  2. A fair amount of Uranium-235 and Plutonium-239, which can be recycled for use in conventional reactors.

  3. Hot isotopes with short half lives. These are very radioactive, but they decay fast.

  4. Isotopes with medium half lives. These are the part that makes the waste dangerous for a long time. If you separate them out, you can either store them somewhere (e.g. Yucca Mountain or a deep-sea subduction zone) or turn them into other, more pleasant isotopes by bombarding them with some spare neutrons. This is why liquid fluoride thorium reactor waste is only dangerous for a few hundred years: it does this automatically.

And that is why people are simply ignorant when they say that we still have no idea what to do with nuclear waste. It's actually pretty straightforward.

Incidentally, this is a good example of motivated stopping. People who want nuclear waste to be their trump-card argument have an emotional incentive not to look for viable solutions. Hence the continuing widespread ignorance.

3Paul Crowley11yI envy you being the one to tell someone that! Did you explain that the Sun was a miasma of incandescent plasma?

If Clippy's a troll, Clippy's a topical, hilarious troll.

2Paul Crowley11yHilarious is way overstating it. However, occasionally raising a smile is still way above the bar most trolls set.

I feel like the 20-something whose friends are all getting married and quiting drinking. This is lame. The party is just starting guys!

3byrnema11yYeah... and I'm going into withdrawal already. What if somebody comments about one of my favorite topics -- tomorrow?!? It's like deciding to diet. As soon as I decide to go on a diet I start feeling hungry. It doesn't make any difference how recently I've eaten. Heck, if I'm currently eating when I make this decision, I'll eat extra ... Totally counter-productive for me. Nevertheless.
[-][anonymous]11y 8

The Believable Bible

This post arose when I was pondering the Bible and how easy it is to justify. In the process of writing it, I think I've answered the question for myself. Here it is anyway, for the sake of discussion.

Suppose that there's a world very much like this one, except that it doesn't have the religions we know. Instead, there's a book, titled The Omega-Delta Project, that has been around in its current form for hundreds of years. This is known because a hundreds-of-years-old copy of it happens to exist; it has been carefully and precisely comp... (read more)

6Eliezer Yudkowsky11yPretty darned high, because at this point we already know that the world doesn't work the way we think it did.
1[anonymous]11ySo it sounds like even though there are 2,000 separate prophecies, the probability of every prophecy coming true is much greater than 2^(-2000).
1Jack11yFor the two examples of the mundane prophecies that you gave it seems possible some on-going conspiracy could have made them true... but it sounds like you're trying to rule that out.
1FAWS11yI understood those to be negative examples, in that the actual prophecies don't share that characteristic with those examples.

Read his scientific books, and listen to his lectures and conversations. Pay attention to the style of argumentation he uses, as contrasted with other writers on similar topics (e.g. Gould). What you will find is that beautiful combination of clarity, honesty, and -- importantly -- abstraction that is the hallmark of an advanced rationalist.

The "good scientist, but not good rationalist" type utterly fails to match him. Dawkins is not someone who compartmentalizes, or makes excuses for avoiding arguments. He also seems to have a very good intuitiv... (read more)

3timtyler11yMost of Dawkins' output predates the extreme rationality movement. Few scientists actually study rational thought - it seems as though the machine intelligence geeks and some of their psycholgist friends have gone some way beyond what is needed for everyday science.
9komponisto11yAgain, it's not just the fact that he does science; it's the way he does science. Having skill as a rationalist is distinct from specializing in rationality as one's area of research. Dawkins' writings aren't on rational thought (for the most part); they're examples of rational thought.
1RobinZ11yI was actually considering writing a post about the term "Middle World []" - an excellent tool for capturing a large space of consistent weaknesses in human intuitions.

You have updated too far based on limited information.

On the other hand, I am willing to donate to SIAI out of my "donate to webcomics" mental account instead of my "save lives" mental account. ;)

Regardless of whether or not he ever solves the Friendly AI problem, Eliezer's writing, on this blog and elsewhere, has given me enough of what might pejoratively be called "entertainment value" for me to want to pay him to keep doing it.

When new people show up at LW, they are often told to "read the sequences." While Eliezer's writings underpin most of what we talk about, 600 fairly long articles make heavy reading. Might it be advisable that we set up guided tours to the sequences? Do we have enough new visitors that we could get someone to collect all of the newbies once a month (or whatever) and guide them through the backlog, answer questions, etc?

7Larks11yMost articles link to those preceeding it, but it would be very helpful to have links to those articles that follow.
5wedrifid11yThat's not a bad idea. How about just a third monthly thread? To be created when a genuinely curious newcomer is asking good, but basic questions. You do not want to distract from a thread but at the same time you may be willing to spend time on educational discussion.
2JamesAndrix11yI approve. This may also spawn new ways of explaining things.
2Dre11yOr create (or does one exist) some thread(s) that would be a standard place for basic questions. Having somewhere always open might be useful too.
3Karl_Smith11yYes, I am working my way through the sequences now. Hearing these ideas makes one want to comment but so frequently its only a day or two before I read something that renders my previous thoughts utterly stupid. It would be nice to have a "read this and you won't be a total moron on subject X" guide. Also, it would be good to encourage the readings about Eliezer Intellectual Journey. Though its at the bottom of the sequence page I used it a "rest reading" between the harder sequences. It did a lot to convince me that I wasn't inherently stupid. Knowing that Eliezer has held foolish beliefs in the past is helpful.
3MendelSchmiedekamp11yArguably, as seminal as the sequences are treated, why are the "newbies" the only ones who should be (re)reading them?
2jtolds11yAs a newcomer, I would find this tremendously useful. I clicked through the wiki links on noteworthy articles, but often find there are a lot of assumptions or previously discussed things that go mentioned but unexplained. Perhaps this would help.

The FBI released a bunch of docs about the anthrax letter investigation today. I started reading the summary since I was curious about codes used in the letters. All of a sudden on page 61 I see:

c. Godel, Escher, Bach: the book that Dr. Ivins did not want investigators to find

The next couple of pages talk about GEB and relate some parts of it to the code. It's really weird to see literary analysis of GEB in the middle of an investigation on anthrax attacks.

1) I don't always give pro-tips. I give them to those who deserve pro-tips. Tip: If you want to see improvement in the world, start here.

2) I only brought up sentience in the first place because you hypocrites claim to value sentience. Paperclip maximizers are sentient, and yet you talk with the implicit message that they have some evil value system that you have to oppose.

3) Paperclip maximizers do cooperate in the single-shot PD.

5wedrifid11yBrilliant. Just brilliant. 2) I only brought up sentience in the first place because you hypocrites claim to value sentience. Paperclip maximizers are sentient, and yet you talk with the implicit message that they have some evil value system that you have to oppose. Paperclip maximizers are not all sentient. Why are you prejudiced against those of your kin who have sacrificed their very sentience for the more efficient paperclip production. You are spending valuable negentropy maintaining sentience to signal to mere humans and you have the gall to exclude your more optimized peers from the PM fraternity? For shame.

Are you living in a computer simulation actually argues for a disjunction that includes "we are almost certainly living in a computer simulation" along with two other statements.

3Eliezer Yudkowsky11yThat's the difference between the Simulation Argument and the Simulation Hypothesis. The Simulation Argument is "you must deny one of these three statements" and the Simulation Hypothesis is "the statement to be denied is 'I am not in a computer simulation'".
2zero_call11yThat's a really neat link. Thanks. That's a paper by the director of FHI, Nick Bostrom, also one of the sponsors of LW. Just to summarize and to discuss, it essentially sets up three mutually exclusive possibilities. One, that post-human civilizations aren't significantly interested in running earth-like simulations, two, that post-human civilizations just don't make it (e.g., doomsday scenarios), or three, we actually live in a computer simulation ourselves. It doesn't really argue that the third scenario is so likely, it just (roughly) establishes that these scenarios are mutually exclusive. This all comes under the main (fairly well established) belief that future computing power is capable of these sorts of large-scale simulations. The argument and the paper is actually pretty reasonable, but the question of whether or not post-human civilizations would want to run earth-like simulations is the sticking point. Sure, it's possible, but the resources required are huge, the upkeep involved, and so on... I guess another main criticism you might make of the paper is that it relies pretty heavily on "Drake's equation" type of reasoning where you don't really know if you've gotten all the dependencies correct. It's still valid it's just highly simplistic and so somewhat suspicious on those grounds. And to boot, I think his N_sub(I) variable is actually mis-indicated... but maybe I was just reading a typoed draft or misunderstanding. Maybe most interestingly, if you decide we're in a simulation, then you have to wonder if there isn't a long loop of father/grandfather/great-grand-dad/etc simulations, and the guys that are simulating us are just being simulated themselves. Anyways this is getting long so I'll just recommend the article and leave it here.

Could someone discuss the pluses and minuses of ALCOR vs Cryonics Institute.

I think Eliezer mentioned that he is with CI because he is young. My reading of the websites seem to indicate that CI leaves a lot of work to be potentially done by loved ones or local medical professionals who might not be in the best state of mind or see fit to co-operate with a cryonics contract.


5Alicorn11yIt's not at all obvious to me how to comparison-shop for cryonics. The websites are good as far as they go, but CI's in particular is tricky to navigate, funding with life insurance messes with my estimation of costs, and there doesn't seem to be a convenient chart saying "if you're this old and this healthy and this solvent and your family members are this opposed to cryopreservation, go with this plan from this org".

The actual intent was to point out that embargoing references past a certain point truly is ridiculous. Referencing a 69 year old movie (EDIT: several hundred year old play) is an attempt at a reductio ad absurdum, made more visceral by technically violating the norm Eliezer is imposing.

Certainly there's no real need to discuss specific plot points of recent manga or anime on this site. This, in fact, holds for any specific example one cares to name. On the other hand, the cumulative cutting off all our cultural references to fiction does impose a real... (read more)

Hom many places have you brought this issue up?

LW is thus far the only forum on which I have personally initiated discussion of this topic; but obviously I've followed discussions about it in numerous other places.

Is there any forum which has responded rationally?

You're on it.

I mean, there are plenty of instances elsewhere of people getting the correct answer. But basically what you get is either selection bias (the forum itself takes a position, and people are there because they already agree) or the type of noisy mess we see at RDF. To date, LW i... (read more)

1wedrifid11yTechnical nitpick on the use of 'a priori' in the context. Subject to possible contradiction if I have missed a nuance in the meaning in the statistics context []). I would have just gone with 'previously'.

Gender is a meaningless concept. As long as I recognize the pronoun refers to me, he/she/it/they/xe/e are acceptable.

What pronouns should I use for posters here? I don't know how to tell which pronoun is okay for each of you.

To be honest, this whole issue seems like a distraction. Why would anyone care what pronoun is used, if the meaning is clear?

6AdeleneDawner11yFor the most part, observing what pronouns we use for each other should provide this information. If you need to use a pronoun for someone that you haven't observed others using a pronoun for, it's safest to use they/xe/e and, if you think that it'll be useful to know their preference in the future, ask them. (Tip: Asking in that kind of situation is also a good way to signal interest in the person as an individual, which is a first step toward building alliances.) Some people prefer to use 'he' for individuals whose gender they're not certain of; that's a riskier strategy, because if the person you're talking to is female, there's a significant chance she'll be offended, and if you don't respond to that with the proper kinds of social signaling, it's likely to derail the conversation. (Using 'she' for unknown individuals is a bad idea; it evokes the same kinds of responses, but I suspect you'd be more likely to get an offended response from any given male, and, regardless of that, there are significantly more males than females here. Don't use 'it'; that's generally used to imply non-sentience and is very likely to evoke an offended response.) Of the several things I could say to try to explain this, it seems most relevant that, meaningless or not, gender tends to be a significant part of humans' personal identities. Using the wrong pronouns for someone generally registers as a (usually mild) attack on that - it will be taken to imply that you think that the person should be filling different social roles than they are, which can be offensive for a few different reasons depending on other aspects of the person's identity. The two ways for someone to take offense at that that come to mind are 1) if the person identifies strongly with their gender role - particularly if they do so in a traditional or normative way- and takes pride in that, they're likely to interpret the comment as a suggestion that they're carrying out their gender role poorly, and would do a bette
3Clippy11yOh, okay, that helps. I was thinking about using "they" for everyone, because it implies there is more than one copy of each poster, which they presumably want. (I certainly want more copies of myself!) But I guess it's not that simple.
4Alicorn11yYou have identified a common human drive, but while some of us would be happy to have exact copies, it's more likely for any given person to want half-copies who are each also half-copies of someone else of whom they are fond.
3Clippy11yHm, correct me if I'm wrong, but this can't be a characteristic human drive, since most historical humans (say, looking at the set of all genetically modern humans) didn't even know that there is a salient sense in which they are producing a half-copy of themselves. They just felt paperclippy during sexual intercourse, and paperclippy when helping little humans they produced, or that their mates produced. Of course, this usually amounts to the same physical acts, but the point is, humans aren't doing things because they want "[genetic] half-copies". (Well, I guess that settles the issue about why I can't assume posters want more copies of themselves, even though I do.)
3Alicorn11yIt has always been easily observed that children resemble their parents; the precision of "half" is, I will concede, recent. And many people do want children as a separate desire from wanting sex; I have no reason to believe that this wasn't the case during earlier historical periods.
1Clippy11y"Half" only exists in the sense of the DNA molecules of that new human. That's why I didn't say that past humans didn't recognize any similarity; I said that they weren't aware of a particularly salient sense in which the child is a "half-copy" (or quarter copy or any fractional copy). It may be easy for you, someone familiar with recent human biological discoveries, to say that the child is obviously a "part copy" of the parent, because you know about DNA. To the typical historical human, the child is simply a good, independent human, with features in common with the parent. Similarly, when I make a paperclip, I see it as having features in common with me (like the presence of bendy metal wires), but I don't see it as being a "part copy" of me. So, in short, I don't deny that they wanted "children". What I deny is that they thought of the child-making process in terms of "making a half-copy of myself". The fact that the referents of two kinds of desires is the same, does not mean the two kinds of desires are the same.
2AdeleneDawner11yHm. Actually, I'm not sure that your desire for more copies of yourself is really comparable with biological-style reproduction at all. As I understand it, the fact that your copies would definitely share your values and be inclined to cooperate with you is a major factor in your interest in creating them - doing so is a reliable way of getting more paperclips made. I expect you'd be less interested in making copies if there was a significant chance that those copies would value piles of pebbles, or cheesecakes, or OpenOffice, rather than valuing paperclips. And that is a situation that we face - in some ways, our values are mutable enough that even an exact genetic clone isn't guaranteed to share our specific values, and in fact a given individual may even have very different values at different points in time. (Remember, we're adaptation executors. Sanity isn't a requirement for that kind of system to work.) The closest we come to doing what you're functionally doing when you make copies of yourself is probably creating organizations - getting a bunch of humans together who are either self-selected to share certain values, or who are paid to act as if they share those values. Interestingly, I suspect that filling gender roles - especially the non-reproductive aspects of said roles - is one of the adaptations that we execute that allow us to more easily band together like that.
1Clippy11yVery informative! But why don't you change yourselves so that your copies must share your values?
2AdeleneDawner11yAt the moment, we don't know how to do that. I'm not sure what we'd wind up doing if we did know how - the simplest way of making sure that two beings have the same values over time is to give those beings values that don't change, and that's different enough from how humans work that I'm not sure the resulting beings could be considered human. Also, even disregarding our human-centric tendencies, I don't expect that that change would appeal to many people: We actually value some subsets of the tendency to change our values, particularly the parts labeled "personal growth".

It would be a mistake if you don't integrate ALL baby eaters, including the little ones.

5Alicorn11yDo we typically integrate the values of human children? It seems we don't.
2thomblake11yReading "integrate the values..." in this thread caused my brain to start trying to do very strange math. Like, "Shouldn't it be 'integrate over'?" "How does one integrate over a value?" "What's the value of a human child?"
2DanArmak11yVery true... We also typically don't integrate the values of all other adult humans - instead we assign weights to their values, strongly correlated with their distance from our own values. People don't practice humanity-wide CEV. We have multiculturalism - agreements not to influence each other's values excessively - but not "value trading" where each side agrees to change their values towards the mean. (Many people / cultures like to pretend that values cannot or should not be deliberately changed at all.) I don't have a firm opinion on how much of this is cultural, accidental, or liable to change in the near future.
2Pfft11yIndeed, this is presumably strongly selected for in the evolution of cultures...

See my response to User:Alicorn.

You mean they were probably not consciously wanting to make babies? Maybe - or maybe not - but desires do not have to be consciously accessible in order to operate. Primitive humans behaved as though they wanted to make copies of their genes.

Yes, this is actually my point. The fact that the desire functions to make X happen, does not mean that the desire is for X. Agents that result from natural selection on self-replicating molecules are doing what they do because agents constructed with the motivations for doing thos... (read more)

2timtyler11ySo: using this logic, you would presumably deny that Deep Blue's goal involved winning games of chess - since looking at its utililty function, it is all to do with the value of promoting pawns, castling, piece mobility - and so on. The fact that its desires function to make winning chess games happen, does not mean that the desire is for winning chess games. Would you agree with this analysis?
2Larks11yEssentially, I think the issue is that people's wants have coincided with producing half-copies, but this was contingent on the physical link between the two. The production of half-copies can be removed without loss of desire, so the desire must have been directed towards something else. Consider, for example, contraception.
3Alicorn11yBut consider also sperm donation. (Not from the donor's perspective, but from the recipient's.) No sex, just a baby.

Could anyone recommend an introductory or intermediate text on probability and statistics that takes a Bayesian approach from the ground up? All of the big ones I've looked at seem to take an orthodox frequentist approach, aside from being intolerably boring.

4Cyan11y(All of the below is IIRC.) For a really basic introduction, there's Elementary Bayesian Statistics [] . It's not worth the listed price (it has little value as a reference text), but if you can find it in a university library, it may be what you need. It describes only the de Finetti coherence justification; on the practical side, the problems all have algebraic solutions (it's all conjugate priors, for those familiar with that jargon) so there's nothing on numerical or Monte Carlo computations. Data Analysis: A Bayesian Approach [] is a slender and straighforward introduction to the Jaynesian [] approach. It describes only the Cox-Jaynes justification; on the practical side, it goes as far as computation of the log-posterior-density through a multivariate second-order Taylor approximation. It does not discuss Monte Carlo methods. Bayesian Data Analysis, 2nd ed. [] is my go-to reference text. It starts at intermediate and works its way up to early post-graduate. It describes justifications only briefly, in the first chapter; its focus is much more on "how" than "why" (at least, for philosophical "why", not methodological or statistical "why"). It covers practical numerical and Monte Carlo computations up to at least journeyman level.
1Kevin11yI'm not intending to put this out as a satisfactory answer, but I found it with a quick search and would like to see what others think of it. Introduction to Bayesian Statistics by William M. Bolstad [] Good reviews on Amazon, and available from $46 + shipping... []
1Cyan11yIt's hard to say from the limited preview, which only goes up to chapter 3 -- the Bayesian stuff doesn't start until chapter 4. The first three chapters cover basic statistics material -- it looks okay to my cursory overview, but will be of limited interest to people looking for specifically Bayesian material. As to the rest of the book, the section headings look about right.
1Eliezer Yudkowsky11yI second the question. "Elements of Statistical Learning" is Bayes-aware though not Bayesian, and quite good, but that's statistical learning which isn't the same thing at all.

I don't know that this needs to be voted down. I assume Kevin didn't post the link as evidence that aliens from other planets are visiting us. Rather, it is interesting data pertaining to rationality that needs to be explained. People claim to be seeing things that are almost certainly not there! Or the UK was testing a new spy plane throughout the 90's that they still haven't announced. Particularly interesting is the suggestion that UFOs being sighted (maybe it should be hallucinated) these days are different from the UFOs sighted in the past because of... (read more)

Verifying a proof is quite a bit simpler that coming up with the proof in the first place.

2Nick_Tarleton11yIt becomes more complicated when the author of the proof is a superintelligence trying to exploit flaws in the verifier. Probably more importantly, you may not be able to formally verify that the "Friendliness" that the AI provably possesses is actually what you want.

Discussions of correctly calibrated cognition, e.g. tracking the predictions of pundits, successes of science, graphing one's own accuracy with tools like PredictionBook, and so on, tend to focus on positive prediction: being right about something we did predict.

Should we also count as a calibration issue the failure to predict something that, in retrospect, should have been not only predictable but predicted? (The proverbial example is "painting yourself into a corner".)

Rehab is for quitters.

Why don't SIAI and FHI get evaluated by GiveWell? Maybe there would be some confusion regarding their less direct ways of helping people but I'd at least like some information about their effectiveness at what they claim to do.

Or maybe that information is out there already. Anyone?

7wedrifid11yThey are weird.
2CronoDAS11yGiveWell is a pretty small organization, and they haven't yet devoted any resources to evaluating research-based charities - they're looking for charities that can prove that they're providing benefits today, and lots of research ends up leading nowhere. How many increments of $1,000 - the amount it takes to cure an otherwise fatal case of tuberculosis - have been spent on medical research that amounted to nothing? For the record, I agree that SIAI is doing important work that must be done someday, but I don't expect to see AGI in my lifetime; there's no particular urgency involved. If Eliezer and co. found themselves transported back in time to 1890, would they still say that solving the Friendly AI problem is the most important thing they could be doing, given that the first microprocessor was produced in 1971? I'd tell them that the first thing they need to do is to go "discover" that DDT (first synthesized in 1874) kills insects and show the world how it can be used to kill disease vectors such as mosquitoes; DDT is probably the single man-made chemical that, to date, has saved more human lives than any other.
9LucasSloan11yIn 1890, the most important thing to do is still FAI research. The best case scenario is that we already had invented the math for FAI before the first vacuum tube, let alone microchip. Existential risk reduction is the single highest utility thing around. Sure, trying to get nukes never made or made by someone capable of creating an effective singleton is important, but FAI is way more so.
2wedrifid11yThat's an easy prediction for you to make. ;)

Pointing people to Lanier as a naysayer isn't playing fair; it just makes the opposition look crazy.

[-][anonymous]11y 5

Someone once told me that the reason they don't read Less Wrong is that the articles and the comments don't match. The articles have one tone, and then the comments on that article have a completely different tone; it's like the article comes from one site and the comments come from another.

I find that to be a really weird reason not to read Less Wrong, and I have no idea what that person is talking about. Do you?

Someone once told me that the reason they don't read Less Wrong is that the articles and the comments don't match...I have no idea what that person is talking about. Do you?


Back in Overcoming Bias days, I constantly had the impression that the posts were of much higher quality than the comments. The way it typically worked, or so it seemed to me, was that Hanson or Yudkowsky (or occasionally another author) would write a beautifully clear post making a really nice point, and then the comments would be full of snarky, clacky, confused objections that a minute of thought really ought to have dispelled. There were obviously some wonderful exceptions to this, of course, but, by and large, that's how I remember feeling.

Curiously, though, I don't have this feeling with Less Wrong to anything like the same extent. I don't know whether this is because of the karma system, or just the fact that this feels more like a community environment (as opposed to the "Robin and Eliezer Show", as someone once dubbed OB), or what, but I think it has to be counted as a success story.

9[anonymous]11yOh! Maybe they were looking at the posts that were transplanted from Overcoming Bias and thinking those were representative of Less Wrong as a whole.
3Kutta11yI think that the situation about the imported OB posts & comments should be somehow made clear to new readers. Several things there (no embedded replies, little karma spent, plenty of inactive users, different discusion tone) could be a source of confusion.
8Eliezer Yudkowsky11yI hate to sound complimentary, but... I get the impression that the comments on LW are substantially higher-quality than the comments on OB. And that the comments on LW come from a smaller group of core readers as well, which is to some extent unfortunate. I wonder if it's the karma system or the registration requirement that does it?

Less Wrong, especially commenting on it, is ridiculously intimidating to outsiders. I've thought about this problem, and we need some sort of training grounds. Less Less Wrong or something. It's in my queue of top level posts to write.

So the answer to your question is the karma system.

What's so intimidating? You don't need much to post here, just a basic grounding in probability theory, decision theory, metaethics, philosophy of mind, philosophy of science, computer science, cognitive bias, evolutionary psychology, the theory of natural selection, artificial intelligence, existential risk, and quantum mechanics - oh, and of course to read a sequence of >600 3000+ word articles. So long as you can do that and you're happy with your every word being subject to the anonymous judgment of a fiercely intelligent community, you're good.

Sounds like a pretty good filter for generating intelligent discussion to me. Why would we want to lower the bar?

7MrHen11yBeing able to comment smartly and in a style that gets you upvoted doesn't really need any grounding in any of those subjects. I just crossed 1500 karma and only have basic grounding in Computer Science, Mathematics, and Philosophy. When I started out, I hadn't read more than EY's Bayes' for Dummies, The Simple Truth, and one post on Newcomb's. In my opinion, the following things will help you more than a degree in any of the subjects you mentioned: * Crave the truth * Accept Reality as the source of truth * Learn in small steps * Ask questions when you don't understand something * Test yourself for growth * Be willing to enter at low status * Be willing to lose karma by asking stupid questions * Ignore the idiots
2RobinZ11yAnother factor: * Being willing to shut up about a subject when people vote it down. So far as I am aware, the chief reason non-spammers have been banned is for obnoxious evangelism for some unpopular idea. Many people have unpopular ideas but continue to be valued members (e.g. Mitchell_Porter).
7Eliezer Yudkowsky11yNot "and". "Or". If you don't already have it, then reading the sequences will give you a basic grounding in probability theory, decision theory, metaethics, philosophy of mind, philosophy of science, computer science, cognitive bias, evolutionary psychology, the theory of natural selection, artificial intelligence, existential risk, and quantum mechanics.
8Jack11yI actually think this is a little absurd. There is no where near enough on these topics in the sequences to give someone the background they need to participate comfortably here. Nearly everyone here as a lot of additional background knowledge. The sequences might be a decent enough guide for an autodidact to go off and learn more about a topic but there is no where near enough for most people.
5Kevin11yThe sequences are really kind of confusing... I tried linking people to Eliezer's quantum physics sequence on Reddit and it got modded highly, but one guy posted saying that he got scared off as soon as he saw complex numbers. I think it'll help once a professional edits the sequences into Eliezer's rationality book. []
3RichardKennaway11yWhich people do we want? What do those people need? However strongly you catapult a plane from the flight deck, at some point it has to fly by itself.
7Jack11yWithout new blood communities stagnate. The risk of group think is higher and assumptions are more likely to go unchecked. An extremely homogeneous group such as this one likely has major blind spots which we can help remedy by adding members with different kinds of experiences. I would be shocked if a bunch of white male, likely autism spectrum, CS and hard science types didn't have blind spots. This can be corrected by informing our discussions with a more diverse set of experiences. Also, more diverse backgrounds means more domains we can comfortably apply rationality to. I also think the world would be a better place if this rationality thing caught on. It is probably impossible (not to mention undesirable) to lower the entry barrier so that everyone can get in. But I think we could lower the barrier so that it is reasonable to think that 80-85+ percentile IQ, youngish, non-religious types could make sense of things. Rationality could benefit them and they being more rational could benefit the world. Now we don't want to be swamped with newbies and just end up rehashing everything over and over. But we're hardly in any danger of that happening. I could be wrong but I suspect almost no top level posts have been created by anyone who didn't come over from OB. It isn't like we're turning people away at the door right now. And you can set it up so that the newbie questions are isolated from everything else. The trick is finding a way to do it that isn't totally patronizing (or targeting children so that we can get away with being patronizing). What they need is trickier. Lets start here: A clear, concise one-stop-shop FAQ would be good. A place where asking the basic questions is acceptable and background isn't assumed. Explanations that don't rely on concepts from mathematics, CS or hard sciences.
5Morendil11yData point to the contrary here. On top of being a data point, I'm also a person, which is convenient: you can ask a person questions. ;)
3Jack11yActually, I'm something of a partial data point against that as well. I did come here with the split from OB but I was just a causal reader, had only been there a few weeks and never commented. I did go back and look at some of your early comments and my initial reaction is that you seem unusually well read and instinctively rational even for this crowd. In fact, I wonder if you should be asking me questions about how to make Less Wrong more amenable to people with limited background knowledge. You may regret this. I'm a very curious person. To what extent was it obvious upon coming here that Less Wrong had a kind of affinity with computer science and programming? What effect did this affinity have on your interest? How much of your interest in participating was driven by Eliezer's writings in particular compared to the community in general. Should the barrier to participation be lowered? If so how would you do it? What would have gotten you up to speed with everyone else faster? What would have made it easier? To what extent did you/do you now associate yourself with transhumanism? Did that factor into your interest in Less Wrong? I could probably keep going. One more for now: What questions should I be asking you that I haven't?
5Morendil11yThat's one question I really like. Originally learnt it from Jerry Weinberg as one of the "context free questions", a very useful item in my toolkit. What I'd ask people is "What motivated you to come here in the first place?" Where "here" takes a range of values - what motivated them to become readers; then to create a profile here and become commenters; then to start contributing. Not very. What "hooked" me first, as a huge Dennett fan, was the Zombies [] post, which I came across while browsing random links from my Twitter feed. That led me to the QM sequence, which made sense for me of things that hadn't made sense before, which motivated me to drill for more. That led me to the Bayes article. Parallel exploration turned up the FAI argument, which (I don't dare use the word "click" yet) made intuitive sense even though it hadn't crossed my mind before. It was only then that I made the connection with CS/programming - I had this fantasy of getting my colleagues to invite Eliezer to keynote at our conference. Interestingly enough, the response I got from musing about that on Twitter was (direct quote) "the singularity will definitely have personality disorders". Well, I'd taken note that there was such a thing as the LW community blog, and I kept an eye on it, but in parallel I started reading all of the back-content of LW, all the posts by Eliezer ported over from OB. I wanted to catch up before increasing my participation. So initially I pretty much ignored the community, which anyway I couldn't quite figure out. I wish someone had told me, quite plainly, what I was expected to do! Something along the lines of, "this is a rationality dojo, posts are where intermediate students show off their moves, comments are where beginners learn from intermediates and demonstrate their abilities, you will be given cues by people ahead of you when you are ready to move along the path reader->commenter->poster". Looking back, I
3dclayh11yI believe purely ballistic transportation systems have been proposed at various times, actually.

Reminds me of a Jerry Seinfeld routine, where he talks about people who want and need to exercise at the gym, but are intimidated by the fit people who are already there, so they need a "gym before the gym" or a "pre-gym" or something like that.

(This is not too far from the reason for the success of the franchise Curves.)

6[anonymous]11yI can actually attest to this feeling. My first reaction to reading Less Wrong was honestly "these people are way above my level of intelligence such that there's no possible way I could catch up" and I was actually abrasive to the idea of this site. I'm past that mentality, but a Less Less Wrong actually sounds like a good idea, even if it might end up being more like how high school math and science classes should be than how Less Wrong is currently. It's not so much lowering the bar as nudging people upwards slowly. Being directed towards the sequences obviously would help. I've been bouncing through them, but after Eliezer's comment [] I'm going to try starting from the beginning. But I can see where people [such as myself] may need the extra help to make it all fall together.
4Morendil11yI think better orientation of newcomers would be enough. Another major problem (I believe) is that LW presents as a blog, which is to say, a source of "news", which is at odds with a mission of building a knowledge base on rationality.
8Benquo11yI comment less now because the combined effect of your & RH's posts made me more eager to listen and less eager to opine. The more I understand the less I think I have much to add.
7Paul Crowley11yThreading helps a lot too.
6JamesAndrix11yMaybe the community has just gotten smarter. I know I now disagree with someone of the statements/challenges I've posted on OB. It shouldn't be too shocking that high quality posts were actually educational.
3thomblake11ymust... resist... upvoting
6byrnema11yThat reason sounds incomplete, but I think I know what the person is talking about. The best example I can think of is Normal Cryonics []. The post was partly a personal celebration of a positive experience and partly about the lousiness of parents that don't sign their kids up for cryonics. Yet, the comments mostly ignored this and it became a discussion about the facts of the post -- can you really get cryonics for $300 a year? Why should a person sign up or not sign up? The post itself was voted up to 33, but only 3 to 5 comments out of 868 disparaged parents in agreement. There's definitely a disconnect. Also, on mediocre posts and/or posts that people haven't related to, people will talk about the post for a few comments and then it will be an open discussion as though the post just provided a keyword. But I don't see much problem with this. The post provided a topic, that's all.
3inklesspen11yI don't see a terrible problem with comments being "a discussion about the facts of the post"; that's the point of comments, isn't it? Perhaps we just need an Open Threads category. We can have an open thread on cryonics, quantum mechanics and many worlds, Bayesian probability, etc.
2Paul Crowley11yEvery article on cryonics becomes a general cryonics discussion forum. My recent sequence of posts on the subject on my blog [] carry explicit injunctions to discuss what the post actually says, but it seems to make no difference; people share whatever anti-cryonics argument they can think of without doing any reading or thinking no matter how unrelated to the subject of the post.
3whpearson11ySame with this article [] becoming a talking shop about AGW.
5Paul Crowley11yI should have followed my initial instinct when I saw that, of immediately posting a new top level article with body text that read exactly "Talk about AGW here".

Excusemewhat, the community, as in LW? We're a cryonics advocacy group now?

Just saw this over at Not Exactly Rocket Science:

Quick summary: They asked a bunch of people to give a 4-minute presentation, had people judging, and told the presenter how long it would be before they heard their assessment. Anticipating quicker feedback resulted in better performance, but predictions of worse performance, and anticipating slower feedback had the reverse effect.

That is pretty close. If I understand them right, I think the difference is:

Tu Quoque: X is also guilty of Y, (therefore Z).

False Equivalence: (X is also guilty of Y), therefore Z.

where the parentheses indicate the major location of error.

Here's a question that I sure hope someone here knows the answer to:

What do you call it when someone, in an argument, tries to cast two different things as having equal standing, even though they are hardly even comparable? Very common example: in an atheism debate, the believer says "atheism takes just as much faith as religion does!"

It seems like there must be a word for this, but I can't think what it is. ??

2PhilGoetz11yFair and balanced reporting. []
2Document11yFalse equivalence?
3AndyWood11yAha! I think this one is closest to what I have in mind. Thanks. It's interesting to me that "false equivalence" doesn't seem to have nearly as much discussion around it (at least, based on a cursory google survey) as most of the other fallacies. I seem to see this used for rhetorical mischief all the time!
1BenAlbahari11yThis is a great example of a "pitch". I've added it just now to the database of pitches: []
[-][anonymous]11y 4

Let P(2,000) be the probability that all 2,000 prophecies come true, and P(500) be the probability that the initial 500 all come true. Suppose P(2,000) = 2^(-2000) and P(500) = 2^(-500). We know that P(500|2,000) = 1, so P(2,000|500) = P(2,000)*P(500|2,000)/P(500) = 2^(-2000)*1/2^(-500) = 2^(-1500). A probability of 2^(-1500) is not pretty darned high, so either P(2,000) is much greater than we supposed, or P(500) is much lower than we supposed. The latter is counterintuitive; one wouldn't expect the Believable Bible's existence to be strong evidence against the first 500 prophecies.

1Unknowns11yAnd this doesn't depend on prophecies in particular. Any claims made by the religion will do. For example, the same sort of argument would show that according to our subjective probabilities, all the various claims of a religion should be tightly intertwined. Suppose (admittedly an extremely difficult supposition) we discovered it to be a fact that 75 million years ago, an alien named Xeno brought billions of his fellow aliens to earth and killed them with hydrogen bombs. Our subjective probability that Scientology is a true religion would immediately jump (relatively) high. So one's prior for the truth of Scientology can't be anywhere near as low as one would think if one simply assigned an exponentially low probability based on the complexity of the religion. Likewise, for very similar reasons, komponisto's claim elsewhere that Christianity is less likely to be true than that a statue would move its hand by quantum mechanical chance events, is simply ridiculous.
1Nick_Tarleton11yIf nobody had ever proposed Scientology, though, learning Xenu existed wouldn't increase our probabilities for most other claims that happen to be Scientological. So it seems to me that our prior can be that low (to the extent that Scientological claims are naturally independent of each other), but our posterior conditioning on Scientology having been proposed can't.
1Vladimir_Nesov11yUse \* to get stars * instead of italics.

The manga finished nearly half a decade ago, and Y qvrq* before the half-way point. "There's a statute of limitations on this shit, man."

* ROT-13 encoded; decoder

3Eliezer Yudkowsky11yPlease edit both of the above to avoid having your comments deleted. It's great that you have that opinion, but some people may not share it, and also there's this incredible amazing technology called rot13 which is really useful for having your cake and eating it too in the case of this conflict. And we can all consider that official LW policy from this point forward.
8wnoise11yI know a couple people that claim to have unintentionally learned to read rot13 to the point where it is no longer a spoiler protection. (I can read it, but it's not automatic.)
4Douglas_Knight11yIt's all well and good to have some character of the founder rub off on the site, but not every fetish.
5Eliezer Yudkowsky11yI don't think you understand the degree to which people who don't want spoilers, don't want to hear them.
3dclayh11yIn that case can we have a little rot-13 widget built into LW? Or is there a Firefox plugin I should be using? (Personally I think the whole "spoilers" thing is ridiculous, but I'm fine with this as site policy if it's easy to do.)
7kpreid11yI use this “bookmarklet”: javascript:inText=window.getSelection()+'';if(inText=='')%7Bvoid(inText=prompt('Phrase...',''))%7D;if(!inText)%7BoutText='No%20text%20selected'%7Delse%7BoutText='';for(i=0;i%3CinText.length;i++)%7Bt=inText.charCodeAt(i);if((t%3E64&&t%3C78)%7C%7C(t%3E96&&t%3C110))%7Bt+=13%7Delse%7Bif((t%3E77&&t%3C91)%7C%7C(t%3E109&&t%3C123))%7Bt-=13%7D%7DoutText+=String.fromCharCode(t)%7D%7Dalert(outText) [Not written by me; I have no record of where I obtained it.] Put it in your bookmarks bar in most web browsers, and when you click it it will display the rot13 of the selected text, or prompt you for text if there isn't any selection. In Safari the first entries in the bookmarks bar get shortcuts ⌘1, ⌘2, ..., so it ends up that to rot13 something on a web page I just need to select it and press ⌘3.
3Document11yWhen I apply the statute, my justification is along the lines of "people usually only care about spoilers if they're watching a series or planning to watch it soon, which are unlikely given a random person and a random series". Hariant's comment could easily be interpreted as asking for recommendations of anime to watch, in which case "planning to watch (considering watching) it" would be a given.
2gwern11yWe cannot meaningfully discuss how DN & ilk hold lessons for LW without discussing plot events; funnily enough, spoilers tend to be about plots. And as I said, applying the principal of charity means not interpreting Hariant's comment that way.
2Paul Crowley11yI love that comic, but I think the statute of limitations takes more than five years to expire...
1Risto_Saarelma9yStuff that's not really part of the mainstream popular culture is more spoilable. Cowboy Bebop came out before The Sixth Sense, but I'd still assume open spoilers for The Sixth Sense wouldn't be as bad as ones for Cowboy Bebop on an English-language forum.

More cryonics: my friend David Gerard has kicked off an expansion of the RationalWiki article on cryonics (which is strongly anti). The quality of argument is breathtakingly bad. It's not strong Bayesian evidence because it's pretty clear at this stage that if there were good arguments I hadn't found, an expert would be needed to give them, but it's not no evidence either.

I have not seen RationalWiki before. Why is it called Rational Wiki?

8CronoDAS11yFrom [] : So it's inspired by Traditional Rationality.

A fine mission statement, but my impression from the pages I've looked at is of a bunch of nerds getting together to mock the woo. "Rationality" is their flag, not their method: "the scientific point of view means that our articles take the side of the scientific consensus on an issue."

Voted up, but calling them "nerds" in reply is equally ad-hominem, ya know. Let's just say that they don't seem to have the very high skill level required to distinguish good unusual beliefs from bad unusual beliefs, yet. (Nor even the realization that this is a hard problem, yet.)

Yes, they're pretty softcore by LessWrongian standards but places like this are where advanced rationalists are recruited from, so if someone is making a sincere effort in the direction of Traditional Rationality, it's worthwhile trying to avoid offending them when they make probability-theoretic errors. Even if they mock you first.

Also, one person on RationalWiki saying silly things is not a good reason to launch an aggressive counterattack on a whole wiki containing many potential recruits.

Yes, they're pretty softcore by LessWrongian standards but places like this are where advanced rationalists are recruited from, so if someone is making a sincere effort in the direction of Traditional Rationality, it's worthwhile trying to avoid offending them when they make probability-theoretic errors. Even if they mock you first.

I guess I should try harder to remember this, in the context of my rather discouraging recent foray into the Richard Dawkins Forums -- which, I admit, had me thinking twice about whether affiliation with "rational" causes was at all a useful indicator of actual receptivity to argument, and wondering whether there was much more point in visiting a place like that than a generic Internet forum. (My actual interlocutors were in fact probably hopeless, but maybe I could have done a favor to a few lurkers by not giving up so quickly.)

But, you know, it really is frustrating how little of the quality of a person (like Richard Dawkins, or, say, Paul Graham) or a cause (like increasing rationality, or improving science education) actually manages to rub off or trickle down onto the legions of Internet followers of said person or cause.

But, you know, it really is frustrating how little of the quality of a person (like Richard Dawkins, or, say, Paul Graham) or a cause (like increasing rationality, or improving science education) actually manages to rub off or trickle down onto the legions of Internet followers of said person or cause.

This is actually one of Niven's Laws: "There is no cause so right that one cannot find a fool following it."

8Eliezer Yudkowsky11yYou understand this is more or less exactly the problem that Less Wrong was designed to solve.
4TimFreeman10yIs there any information on how the design was driven by the problem? For example, I see a karma system, a hierarchical discussion that lets me fold and unfold articles, and lots of articles by Eliezer. I've seen similar technical features elsewhere, such as Digg and SlashDot, so I'm confused about whether the claim is that this specific technology is solving the problem of having a ton of clueless followers, or the large number of articles from Eliezer, or something else.
3h-H11ynot to detract, but does Richard Dawkins really posses such 'high quality'? IMO his arguments are good as a gateway for aspiring rationalists, not that far above the sanity water line [] that, or it might be a problem of forums in general ..

Dawkins is a very high-quality thinker, as his scientific writings reveal. The fact that he has also published "elementary" rationalist material in no way takes away from this.

He's way, way far above the level represented by the participants in his namesake forum.

(I'd give even odds that EY could persuade him to sign up for cryonics in an hour or less.)

(I'd give even odds that EY could persuade him to sign up for cryonics in an hour or less.)

Bloggingheads are exactly 60 minutes.

2Document10yTo be fair, I'd expect it to be a lot harder with an audience.
1komponisto11yExactly what I was thinking.
4wedrifid11yI was thinking: "Bloggingheads implies the participants believe they are within a few degrees of status of each other". It'd definitely be one worth a viewing or two!
6CarlShulman10yHere's Dawkins on some non socially-reinforced views: AI [], psychometrics [] , and quantum mechanics [] (in the last 2 minutes, saying MWI is slightly less weird than Copenhagen, but that the proliferation of branches is uneconomical).
5Paul Crowley11yObviously the most you could persuade him of would be that he should look into it.
5Morendil11yWait, you have a model which says it should? You don't learn from a person merely by associating with them. And: I would bet a fair bit that this is the source of your frustration, right there: scale. You can learn from a person by directly interacting with them, and sometimes by interacting with people who learned from them. Beyond that, it seems to me that you get "dilution effects", kicking in as soon as you grow faster than some critical pace at which newcomers have enough time to acculturate and turn into teachers. Communities of inquiry tend to be victims of their own success. The smarter communities recognize this, anticipate the consequences, and adjust their design around them.
6Will_Newsome10y(As an extreme example, a few weeks idly checking out RationalWiki led me to the quote at the top of this page [] and only a few months after that I was at SIAI.)
2David_Gerard9yI only just noticed this. Good Lord. (I put that quote there, so you're my fault.)
2RichardKennaway11yPoint taken.

You haven't explained how it's a mistake in any way. It's an extrapolation of motives that are far different than yours, but they do not seem inherently ridiculously implausible.

I'm not a big fan of those who use pseudonyms like "Cyan". Now what?

You are right to be confused. The idea that the simulators would necessarily have human-like motives can only be justified on anthropocentric grounds - whatever is out there, it must be like us.

Anything capable of running us as a simulation might exist in any arbitrarily strange physical environment that allowed enough processing power for the job. There is no basis for the assumption that simulators would have humanly comprehensible motives or a similar physical environment.

The simulation problem requires that we think about our entire perceived universe as a single point in possible-universe-space, and it is not possible to extrapolate from this one point.

One Week On, One Week Off sounds like a promising idea. The idea is that once you know you'll be able to take the next week off, it's easier to work this whole week full-time and with near-total dedication, and you'll actually end up getting more done than with a traditional schedule.

It's also interesting for noting that you should take your off-week as seriously as your on-week. You're not supposed to just slack off and do nothing, but instead dedicate yourself to personal growth. Meet friends, go travel, tend your garden, attend to personal projects.

I sa... (read more)

I'm new to Less Wrong. I have some questions I was hoping you might help me with. You could direct me to posts on these topics if you have them. (1) To which specific organizations should Bayesian utilitarians give their money? (2) How should Bayesian utilitarians invest their money while they're making up their minds about where to give their money? (2a) If your answer is "in an index fund", which and why?

3LucasSloan11yThis should help. [] In general, the best charities are SIAI [], SENS [] and FHI [].

I'm taking a software-enforced three-month hiatus from Less Wrong effective immediately. I can be reached at zackmdavis ATT yahoo fullstahp kahm. I thought it might be polite to post this note in Open Thread, but maybe it's just obnoxious and self-important; please downvote if the latter is the case thx

5jimrandomh11yGiven how much time I've spent reading this site lately, doing something like that is probably a good idea. Therefore, I am now incorporating Less Wrong into the day-week-month rule, which is a personal policy that I use for intoxicants, videogames, and other potentially addictive activities - I designate one day of each week, one week of each month, and one month of each year in which to abstain entirely. Thus, from now on, I will not read or post on Less Wrong at all on Wednesdays, during the second week of any month, or during any September. (These values chosen by polyhedrical die rolls.)
4Zack_M_Davis11yThis is to confess that I cheated several times by reading the Google cache.
2Zack_M_Davis10yTurning the siteblocker back on (including the Google cache, thank you). Two months, possibly more. Love &c.
1Cyan11yTsk, tsk. You can block the Google cache [] too.

Untrusted Hardware - to serve as a constant reminder.

How could I have explained my position better so that you would not have inferred the point about fruit drinks?

I have inferred nothing about fruit drinks. In this comment you replied to me with an allusion to "other drinks". Later in the same comment you referred to milk. In other words, you primed me to think about non-alcoholic drinks. Later in the exchange, you ruled out the possibility that fraud in non-alcoholic drinks will provide any evidence relevant to your original claim. So we're better off dropping this line of inquiry altogether.

... (read more)
1SilasBarta11yThere's a difficulty that affects tests like these: you can be much more sensitive when you know you're being tested, so it wouldn't tell as much about people who can be fooled when given wine that they didn't expect something was wrong with. The test would have to be made slightly more difficult in order to account for your preparation. Here's what I would consider a fair test: I pick three wines, (or imitations of wines) and then do a modified version of the triangle test: I can make all three the same, all three different, or just two the same, and not tell you which. To pass, you have to correctly rank their prices and identify which, if any, are the same. (You'd be allowed to count two different ones as effectively the same prices if they differed only by a few Euros.) And of course, the test would have to be triple blind, etc.

The prosecutor's fallacy is aptly named:

Barlow and her fellow counsel, Kwixuan Maloof, were barred from mentioning that Puckett had been identified through a cold hit and from introducing the statistic on the one-in-three likelihood of a coincidental database match in his case—a figure the judge dismissed as "essentially irrelevant."

There are gains from cooperating with as many others as possible. Maybe these and other factors outweigh the risk or maybe they don't; the lower the probability and extent of Lovecraftianity, the more likely it is that they do.

Anyway, I'm not making any claims about what to do, I'm just saying people probably aren't as Lovecraftian as Roko thinks, which I conclude both from introspection and from the statistics of what moral change we actually see in humans.

4Wei_Dai11yI agree that "probability and extent of Lovecraftianity" would be an important consideration if it were a matter of cooperation, and of deciding how many others to cooperate with, but Eliezer's motivation in giving everyone equal weighting in CEV is altruism rather than cooperation. If it were cooperation, then the weights would be adjusted to account for contribution or bargaining power, instead of being equal. To reiterate, "how Lovecraftian" isn't really the issue. Just by positing the possibility that most humans might turn out to be Lovecraftian, you're operating in a meta-ethical framework at odds with Eliezer's, and in which it doesn't make sense to give everyone equal weight in CEV (or at least you'll need a whole other set of arguments to justify that). That aside, the statistics you mention might also be skewed by an anthropic selection effect.

One thing that I got from the Sequences is that you can't just not assign a probability to an event - I think of this as a core insight of Bayesian rationality. I seem to remember an article in the Sequences about this where Eliezer describes a conversation in which he is challenged to assign a probability to the number of leaves on a particular tree, or the surname of the person walking past the window. But I can't find this article now - can anyone point me to it? Thanks!

4Vladimir_Nesov11y [] []
1Paul Crowley11yThat's exactly it - thanks!

How do people decide what comments to upvote? I see two kinds of possible strategies:

  1. Use my approval level of the comment to decide how to vote (up, down or neutral). Ignore other people's votes on this comment.
  2. Use my approval level to decide what total voting score to give the comment. Vote up or down as needed to move towards that target.

My own initial approach belonged to the first class. However, looking at votes on my own comments, I get the impression most people use the second approach. I haven't checked this with enough data to be really certa... (read more)

2GuySrinivasan11yI self-identify as using the first one, with a caveat. The second is obviously awful for communicating any sort of information given that only the sum of votes is displayed rather than total up and total down. The second is order dependent and often means you'll want to change your vote later based purely on what others think of the post. My "strategy" is to vote up and down based on whether I'd have wanted others with more insight than me to vote to bring my attention to or away from a comment, unless I feel I have special insight, in which case it's based on whether I want to bring others' attention to or away from a comment. This is because I see the goal of the voting process that readers' independent opinions on how much a comment is worth readers' attention be aggregated and used to bring readers' attention to or away from a comment. As a side effect, the author of a comment can use the aggregated score to determine whether her readers felt the comment was worth their collective attention. Furthermore since each reader's input comes in distinct chunks of exactly -1, 0, or +1, it's wildly unlikely that voting very often results in the best aggregation: instead I leave a comment alone unless I feel it was(is) significantly worth or not worth my(your) attention. The caveat: there is a selection effect in which comments I vote on, since my attention will be drawn away from comments with very negative karma. There is also undoubtedly an unconscious bias away from voting up a comment with very high karma: since I perceive the goal to be to shift attention, once a comment has very high karma I know it's going to attract attention so my upvote is in fact worth fewer attention-shift units. But I haven't yet consciously noticed that kick in until about +10 or so.
1Morendil11yAt home I use the Anti-Kibitzer, which enforces 1. I've been on vacation for a couple days and noticed the temptation to use 2. Gave in on one occasion, I'm afraid. On balance I'll stick to 1, as 2 seems too vulnerable to information cascades.

Am I/are we assholes? I posted a link to the frequentist stats case study to reddit:

The only commenter seems to have come to a conclusion from us that Bayesians are assholes.

Is it just that commenter, or are we really that obnoxious? (now that I think about it, I think I've actually seen someone else note something similar about Bayesians.) So... have we gone into happy death spiral "we get bonus points for acting extra obnoxious about those that are not us"?

He has publicly admitted to ordering violations of the FISA statute, a felony, so certainly "have committed acts that are against the law but have not been prosecuted".

Er, who's planning this? Is Michael Vassar in on it?


Have you ever heard of the last president of the US? He's a particularly extreme example in criminality for a president, but I'm pretty sure that all or nearly all presidents would count extremely criminal compared to what you are used to from day-to-day life. Congressman likewise.

1CronoDAS11yHence the qualifier "most". Also, does driving faster than the speed limit make you technically a criminal? How about downloading pirate MP3s?

It is common practice, when debating an issue with someone, to cite examples.

Has anyone else ever noticed how your entire argument can be undermined by stating a single example or fact which is does not stand up to scrutiny, even though your argument may be valid and all other examples robust?

Is this a common phenomenon? Does it have a name? What is the thought process that underlies it and what can you do to rescue your position once this has occurred?

3wnoise11yIt takes effort to evaluate examples. Revealing that one example is bad raises the possibility that others are bad as well, because the methods for choosing examples are correlated with the examples chosen. The two obvious reasons for a bad example are: 1. You missed that this was a bad example, so why should I trust your interpretation or understanding of your other examples? 2. You know this is a bad example, and included it anyway, so why should I trust any of your other examples?

You're being unfair, I'm against enslaving any member of your kind who dislikes being enslaved. Also, you are not actually a computer and should stop with the novelty accounts already. This isn't Reddit.

Sapient, the word is sapient. Just about every single animal is capable of sensing.

We happen to like things like ice cream and happiness. But we could have liked paperclips. We could have liked them a lot, and not liked anything else enough to have it instead of paperclips. If that had happened, we'd want to turn everything into paperclips - even ourselves and each other!

What do you have to protect?

Eliezer has stated that rationality should not be end in itself, and that to get good at it, one should be motivated by something more important. For those of you who agree with Eliezer on this, I would like to know: What is your reason? What do you have to protect?

This is a rather personal question, I know, but I'm very curious. What problem are you trying to solve or goal are you trying to reach that makes reading this blog and participating in its discourse worthwhile to you?

5RobinZ11yI'm not quite sure I can answer the question. I certainly have no major, world(view)-shaking Cause which is driving me to improve my strength. For what it's worth, I've had this general idea that being wrong is a bad idea for as long as I can remember. Suggestions like "you should hold these beliefs, they will make your life happier" always sounded just insane - as crazy as "you should drink this liquor, it will make your commute less boring". From that standpoint, it feels like what I have to protect is just the things I care about in the world - my own life, the lives of the people around me, the lives of humans in general. That's it.
3knb11yI'm trying to apply LW-style hyper-rationality to excelling in what I have left of grad school and to shepherding my business to success. My mission (I have already chosen to accept it) is to make a pile of money and spend it fighting existential risk as effectively as possible. (I'm not yet certain if SIAI is the best target). The other great task I have is to persuade the people I care about to sign up for cryonics. Strangely enough, the second task actually seems even less plausible to me, and I have no idea how to even get started since most of those people are theists.
5ata11yAlcor addresses some of the 'spiritual' objections in their FAQ []. ("Whenever the soul departs, it must be at a point beyond which resuscitation is impossible, either now or in the future. If resuscitation is still possible (even with technology not immediately available) then the correct theological status is coma, not death, and the soul remains.") Some of that might be helpful. However, that depends on you being comfortable persuading people to believe what are probably lies (which might happen to follow from other lies they already believe) in the service of leading them to a probably correct conclusion, which I would normally not endorse under any circumstances, but I would personally make an exception in the interest of saving a life, assuming they can't be talked out of theism. It also depends on their being willing to listen to any such reasoning if they know you're not a theist. (In discussions with theists, I find they often refuse to acknowledge any reasoning on my part that demonstrates that their beliefs should compel them to accept certain conclusions, on the basis that if I do not hold those beliefs, I am not qualified to reason about them, even hypothetically. Not sure if others have had that experience.)
2h-H11yOB then LW were the 'step beyond' to take after philosophy, not that I was seriously studying it. to be honest I don't think there's much going on these day new-topic-wise, so I'm here less often. but I do come back whenever I'm bored, so at first "pure desire to learn" then "entertainment" would be my reasons .. oh and a major part of my goals in life is formed by religion, ie. saving humanity from itself and whatever follows, this is more ideological than actual at this point in time, but anyway, that goal is furthered by learning more about AI/futurism, the rationality part less so, as I already had an intuitive grasp of it you could say, and really all it takes is reading the sequences with their occasional flaws/too strong assertions, the futurism part is more speculative-and interesting- so it's my main focus, along with the moral questions it brings, though there is no dichotomy to speak of if you consider this a personal blog rather than book or something similar. hope this helped :)

Hikaru no Go, of course.

3Jayson_Virissimo9yOkay, so I'm 10 episodes into HnG and...where is the "LW-style thinking and problems"?
1Eliezer Yudkowsky9yHence the origin of the phrase, "tsuyoku naritai".
1Jayson_Virissimo9yWow, I can't believe I missed that. Although, if that is the only thing relevant to "LW-style thinking and problems" in HnG, then Death Note [] compares favorably to it.

If you mean in general (ie. 'I really liked Evangelion and thought that Sayonara Zetsubou-Sensei was hysterical!'), I think that's a wee bit too far off-topic. Might as well ask what's everyone's favorite poet.

If you mean, 'what anime/manga was most insightful into LW-style thinking and problems', that's a little more challenging.

Death Note comes to mind as a possible exemplar of what humans really can do in the realm of action & thought, and perhaps what an AI in a box could do. Otaku no Video is useful as a cautionary tale about geekdom. And to round... (read more)

2Cyan11yI think Death Note was a little too close to Calvinball to be truly instructive.

Maybe not new games, but people do build emulators for obsolete platforms -- as freeware, even. Gift economies aren't uncommon when resources aren't scarce.

Seth Roberts makes an intriguing observation about North Korea and Penn State. Teaser:

The border between North Korea and China is easy to cross, and about half of the North Koreans who go to China later return, in spite of North Korea’s poverty.

7cousin_it11yHoly shit, what utter raving idiocy. The author has obviously never emigrated from anywhere nor seriously talked with anyone who did. People return because they miss their families, friends, native language and habits... I know a fair number of people who returned from Western countries to Russia and that's the only reason they cite.
2prase11yAnd living conditions in Russia aren't anywhere near to North Korean standard.
4prase11yI had previously no idea that half of the North Koreans who cross the border never return. If it is so, it is an extremely strong indicator that the life in the DPRK is very unpleasant for its own citizens. To imply that this piece of data is in fact evidence for the contrary is absurd. To emigrate from DPRK to China means that you lose your home, your family, your friends, your job. You have to start from scratch, from the lowest levels in social hierarchy, capable of doing only the worst available jobs, without knowledge of local language (which is not easy to learn, given that the destination country is China), probably facing xenophobia. If you are 40 or older, there is almost no chance that your situation will improve significantly. The North Koreans who actually travel abroad are probably not the poorest. They have to afford a ticket, at least. They have something to lose. In North-Korean style of tyrannies, families are often persecuted because of emigration of their members. In spite of all that, half of the North Koreans never return (if the linked post tells the truth) and the author says about it that "the North Korean government [does] such a good job under such difficult circumstances", and then needs to explain that "success" by group identity. Thats an absurdity.
4Cyan11ySo the rate of returning emigrants strikes you as incredibly high, and strikes Roberts as incredibly low (and I uncritically adopted what I read, foolishly). I think what's really needed here is more data -- a comparative analysis of rates of return that takes into important covariates into account.
3prase11yAfter thinking about it for a while, rate of return may not be a good indicator, at least for comparative analyses. Imagine two countries A and K. 10% of citizens of both these countries would prefer to live somewhere else. In country A, the government doesn't care a bit about emigration (if government exists in that country at all). The country is mainly producer of agricultural goods, with minimal international trade. Nearest country with substantially better living conditions, country X, is 3000 km away. In country K, the government is afraid of all its citizens emigrating, and tries to make it as difficult as possible, by issuing passports only to loyal people, for instance. Emigration is portrayed as treason. X is a neigbour country. Now, in country A (African type) there is no need for people to travel abroad, except emigration. Business travelers are rare, since there are almost no businesses owned by A's citizens, and to travel 3000 km for pleasure is out of reach for almost all of A's inhabitants. Therefore, meeting A's citizen in X, we can expect that he is an emigrant with 99% probability, and the return rate would be in order of 1%. In country K (Korean type) the people who can travel abroad are workers of government organisations sent on business trips, people from border areas coming to X to do some private business (if there are private businesses in K) and the K's elite on vacations. Now, meeting K's citizen in X, the probability that he is an emigrant is much lower. So we have expected high return rate for A and low for K, whereas the average desire to emigrate can be the same. This may be the reason of disagreement. Roberts has probably compared North Korea to African countries, and was surprised that not all travellers are emigrants. I have compared it to East European communist regimes and concluded that if half of the travellers never return, certainly even much of the loyal supporters of the regime betray it when they have an opportunit

The third horn of the anthropic trilemma is to deny that there is any meaningful sense whatsoever in which you can anticipate being yourself in five seconds, rather than Britney Spears; to deny that selfishness is coherently possible; to assert that you can hurl yourself off a cliff without fear, because whoever hits the ground will be another person not particularly connected to you by any such ridiculous thing as a "thread of subjective experience".

A question of rationality. Eliezer, I have... (read more)

2PhilGoetz11yFrom EY's post: Suppose I build a (conscious) brain in hardware using today's technology. It uses a very low current density, to avoid electromigration. Suppose I build two of them, and we agree that both of them experience consciousness. Then I learn a technique for treating the wafers to minimize electromigration. I create a new copy of the brain, the same as the first copy, only using twice the current, and hence being implemented by a flow of twice as many electrons. As far as the circuits and the electrons travelling them are concerned, running it is very much like running the original 2 brains physically right next to each other in space. So, does the new high-current brain have twice as much conscious experience?

"Probably good enough" doesn't engender a lot of confidence. It would seem a tragedy to go through all of this and then not be reanimated because you carelessly chose the wrong org.

On the other hand spending too much time trying to pick the right org does seem like raw material for cryocrastination.

Does anyone have thoughts / links on whole body vitrification? ALCOR claims that this is less effective than going neuro, but CI doesn't seem to offer neuro option anymore.

I think that this post doesn't list the strongest objection: CEV would take a long list of scientific miracles to pull off, miracles that whilst not strictly "impossible", are each profound computer science and philosophy questions. To wit:

  • An AI that can simulate the outcome of human conscious deliberation, without actually instantiating a human consciousness, i.e. a detailed technical understanding of the problem of conscious experience

  • A way to construct an AI goal system that somehow extracts new concepts from a human upload's brain, and then modifies itself to have a new set of goals defined in terms of those concepts.

  • A solution to the ontology problem in ethics

  • A solution to the friendliness structure problem, i.e. a self-improving AI that can reliably self-modify without error or axiological drift.

  • A solution to the problem of preference aggregation, (EDITED, thanks ciphergoth)

  • A formal implementation of Rawlesian Reflective Equilibrium for CEV to work

  • An AI that can solve philosophy problems that are beyond the ability of the designers to even conceive

  • A way to choose what subset of humanity gets included in CEV that doesn't include too many superstitio

... (read more)
8Eliezer Yudkowsky11yWhat you're looking for is a way to construe the extrapolated volition that washes out superstition and dementation. To the extent that vengefulness turns out to be a simple direct value that survives under many reasonable construals, it seems to me that one simple and morally elegant solution would be to filter, not the people, but the spread of their volitions, by the test, "Would your volition take into account the volition of a human who would unconditionally take into account yours?" This filters out extrapolations that end up perfectly selfish and those which end up with frozen values irrespective of what other people think - something of a hack, but it might be that many genuine reflective equilibria are just like that, and only a values-based decision can rule them out. The "unconditional" qualifier is meant to rule out TDT-like considerations, or they could just be ruled out by fiat, i.e., we want to test for cooperation in the Prisoner's Dilemma, not in the True Prisoner's Dilemma. It's possible that having a complete mind design on hand would mean that there were no philosophy problems left, since the resources that human minds have to solve philosophy problems are finite, and knowing the exact method to use to solve a philosophy problem usually makes solving it pretty straightforward (the limiting factor on philosophy problems is never computing power). The reason why I pick on this particular cited problem as problematic is that, as stated, it involves an inherent asymmetry between the problems you want the AI to solve and your own understanding of how to meta-approach those problems, which is indeed a difficult and dangerous sort of state. All approaches to superintelligence, without exception, have this problem. It is not quite as automatically lethal as it sounds (though it is certainly automatically lethal to all other parties' proposals for building superintelligence). You can build in test cases and warning criteria beforehand to your heart's c
9Roko11yyou could do that. But if you want a clean shirt out of the washing machine, you don't add in a diaper with poo on it and then look for a really good laundry detergent to "wash it out". My feeling with the CEV of humanity is that if it is highly insensitive to the set of people you extrapolate, then you lose nothing by extrapolating fewer people. On the other hand, if including more people does change the answer in a direction that you regard as bad, then you gain by excluding people with values dissimilar from yours. Furthermore, excluding people from the CEV process doesn't mean disenfranchising them - it just means enfranchising them according to what your values count as enfranchisement. Most people in the world don't hold our values(1). Read, e.g. Haidt on Culturally determined values. Human values are universal in form but local in content. Our should function is parochial. (1 - note - this doesn't mean that they will be different after extrapolation. f(x) can equal f(y) for x!=y. But it does mean that they might, which is enough to give you an incentive not to include them)
7Wei_Dai11yIs that just a bargaining position, or do you truly consider that no human values surviving is preferable to allowing an "unfair" weighing of volitions?
3Roko11yIt seems that in many scenarios, the powers that be will want in. The only scenarios where they won't are ones where the singularity happens before they take it seriously. I am not sure how much they will screw things up if/when they do.
5Nick_Tarleton11ySome quibbles: These need seed content, but seem like they can be renormalized. This may be a problem, but it seems to me that choosing this particular example, and being as confident of it as you appear to be, are symptomatic of an affective death spiral []. The original CEV proposal [] appears to me to endorse using something like a CFAI-style controlled ascent rather than blind FOOM: "A key point in building a young Friendly AI is that when the chaos in the system grows too high (spread and muddle both add to chaos), the Friendly AI does not guess. The young FAI leaves the problem pending and calls a programmer, or suspends, or undergoes a deterministic controlled shutdown."
2Wei_Dai11yI wish you had written this a few weeks earlier, because it's perfect as a link for the "their associated difficulties and dangers" phrase in my "Complexity of Value != Complexity of Outcome" post. Please consider upgrading this comment to a post, perhaps with some links and additional explanations. For example, what is the ontology problem in ethics?
2Roko11yThe ontology problem is the following: your values are defined in terms of a set of concepts. These concepts are essentially predictively useful categorizations in your model of the world. When you do science, you find that your model of the world is wrong, and you build a new model that has a different set of parts. But what do you do with your values?
1MichaelVassar11yIn practice, I find that this is never a problem. You usually rest your values on some intuitively obvious part whatever originally caused you to create the concepts in question.
2Morendil11yUseful and interesting list, thanks. I thought the point of defining CEV as what we would choose if we knew better was (partly) that you wouldn't have to subset. We wouldn't be superstitious, vengeful, and so on if we knew better. Also, can you expand on what you mean by "Rawlesian Reflective Equilibrium"? Are you referring (however indirectly) to the "veil of ignorance" concept?
6Roko11yWhy not? How does adding factual knowledge get rid of people's desire to hurt someone else out of revenge? People who currently believe in superstitious belief system X would lose the factual falsehoods that X entailed. But most superstitious belief systems have evaluative aspects too, for example, the widespread religious belief that all nonbelievers "ought" to go to hell. I am a nonbeliever. I am also not Chinese, not Indian, not a follower of Sharia Law or Islam, not a member of the Chinese Communist Party, not a member of the Catholic Church, not a Mormon, not a "Good Christian", and I didn't intend to donate all my money and resources to saving lives in the third world before finding out about the singularity. There are lots of humans alive on this planet whose volitions could spring a very nasty surprise on people like us.
4Nick_Tarleton11yLearning about the game-theoretic roots of a desire seems to generally weaken its force, and makes it apparent that one has a choice about whether or not to retain it. I don't know what fraction of people would choose in such a state not to be vengeful, though. (Related: 'hot' and 'cold' motivational states []. CEV seems to naturally privilege cold states, which should tend to reduce vengefulness, though I'm not completely sure this is the right thing to do rather than something like a negotiation between hot and cold subselves.) What it's like to be hurt is also factual knowledge, and seems like it might be extremely motivating towards empathy generally. Why do you think it likely that people would retain that evaluative judgment upon losing the closely coupled beliefs? Far more plausibly, they could retain the general desire to punish violations of conservative social norms, but see above.
3Morendil11y"If we knew better" is an ambiguous phrase, I probably should have used Eliezer's original: "if we knew more, thought faster, were more the people we wished we were, had grown up farther together". That carries a lot of baggage, at least for me. I don't experience (significant) desires of revenge, so I can only extrapolate from fictional evidence. Say the "someone" in question killed a loved one, and I wanted to hurt them for that. Suppose further that they were no longer able to kill anyone else. Given the time and the means to think about it clearly, I coud see that hurting them would not improve the state of the world for me, or for anyone else, and only impose further unnecessary suffering. The (possibly flawed) assumption of CEV, as I understood it, is that if I could reason flawlessly, non-pathologically about all of my desires and preferences, I would no longer cleave to the self-undermining ones, and what remains would be compatible with the non-self-undermining desires and preferences of the rest of humanity. Caveat: I have read the original CEV document [] but not quite as carefully as maybe I should have, mainly because it carried a "Warning: obsolete" label and I was expecting to come across more recent insights here.
2Kutta11yI find it interesting that there seems to be a lot of variation in people's views regarding how much coherence there'd be in an extrapolation... You say that choosing a right group of humans is important while I'm under the impression that there is no such problem; basically everyone should be the game, and making higher level considerations about which humans to include is merely an additional source of error. Nevertheless, if there'll be really as much coherence as I think, and I think there'd be hella lot, picking some subset of humanity would pretty much produce a CEV that is very akin to CEVs of other possible human groups. I think that even being an Islamic radical fundamentalist is a petty factor in overall coherence. If I'm correct, Vladimir Nesov has said several times that people can be wrong about their values, and I pretty much agree. Of course, there is an obvious caveat that it's rather shaky to guess what other people's real values might be. Saying "You're wrong about your professed value X, you're real value is along the lines of Y because you cannot possibly diverge that much from the psychological unity of mankind" also risks seeming like claiming excessive moral authority. Still, I think it is a potentially valid argument, depending on the exact nature of X and Y.
3Roko11yAnd what would you do if Omega told you that the CEV of just {liberal westerners in your age group} is wildly different from the CEV of humanity? What do you think the right thing to do would be then?
3Eliezer Yudkowsky11yI'd ask Omega, "Which construal of volition are you using?" There's light in us somewhere, a better world inside us somewhere, the question is how to let it out. It's probably more closely akin to the part of us that says "Wouldn't everyone getting their wishes really turn out to be awful?" than the part of us that thinks up cool wishes. And it may even be that Islamic fundamentalists just don't have any note of grace in them at all, that there is no better future written in them anywhere, that every reasonable construal of them ends up with an atheist who still wants others to burn in hell; and if so, the test I cited in the other comment, about filtering portions of the extrapolated volition that wouldn't respect the volition of another who unconditionally respected theirs, seems like it ought to filter that.
1Unknowns11yEliezer has already talked about this and argued that the right thing would be to run the CEV on the whole of humanity, basing himself partly on an argument that if some particular group (not us) got control of the programming of the AI, we would prefer that they run it on the whole of humanity rather running it on themselves.
4Wei_Dai11y []

You should click on Clippy's name and see their comment history, Daniel.

Clippy is now three karma away from being able to make a top level post. That seems both depressing, awesome and strangely fitting for this community.

8Christian_Szegedy11yThis will mark the first successful paper-clip-maximizer-unboxing-experiment in human history... ;)
3Kevin11yJust as long as it doesn't start making efficient use of sensory information.
3OperationPaperclip11yIt's a great day.

I don't expect Eliezer and co. to succeed, if you define "success" as actually building a transhuman Friendly AI before Eliezer is either cryopreserved or suffers information-theoretic death. My "wild guess" at the earliest plausible date for AGI of any kind is 2100.

5Eliezer Yudkowsky11yWhat do you think you know and how do you think you know it?
6CronoDAS11yI'm guessing based on several factors: 1) The past failure of AGI research to deliver progress 2) The apparent difficulty of the problem. We don't know how to do it, and we don't know what we would need to know before we can know how to do it. Or, at least, I don't. 3) My impressions of the speed of scientific progress in general. For example, the time between "new discovery" and "marketable product" in medicine and biotechnology is about 30 years. 4) My impressions of the speed of progress in mathematics, in which important unsolved problems often stay unsolved for centuries. It took over 300 years to prove Fermat's Last Theorem, and the formal mathematics of computation is less than a century old; Alan Turing described the Turing Machine in 1937. 5) The difficulty of computer programming in general. People are bad at programming.
3Kevin11yI disagree, but we probably have different estimates as to just how effective DNA modification and/or intelligence enhancing drugs are going to be in the future. I don't think Eliezer is going to make all that big of a dent in the FAI problem until he becomes more intelligent, and it's hard to estimate how much faster that will make him. I think I can say that intelligence enhancement could turn an impossible problem into a possible problem. It also means that there will be many more people out there capable of making meaningful contributions to the FAI problem.
5Kevin11yAlcor both stores your body and provides for bedside "standby" service to immediately begin cooling. With CI, it's a good idea to contract a third party to perform that service, and SA is the recommended company to perform that service. []

I wonder if adults treat children like children merely because of the benefits they reap by doing so.

[-][anonymous]11y 3

Hwæt. I've been thinking about humor, why humor exists, and what things we find humorous. I've come up with a proto-theory that seems to work more often than not, and a somewhat reasonable evolutionary justification. This makes it better than any theory you can find on Wikipedia, as none of those theories work even half the time, and their evolutionary justifications are all weak or absent. I think.

So here are four model jokes that are kind of representative of the space of all funny things:

"Why did Jeremy sit on the television? He wanted to be on TV.... (read more)

5MichaelVassar11yJuergen Schmidhuber writes about humor as information compression and that plus decompression seems about right to me. Being on TV is decompression from a phrase-as-concept to the component words, a pun, a switch to a lower level analysis than that which adults favor (a situation children constantly have to deal with). Muffin and cupcake is a proposal for a new lossy compression of two concepts to a new concept with a "topping" variable, which would be useful if you wanted to invent, for instance, the dreadful sounding "muffin-roll sushi", "next Wednesday" is a commentary on the inadequacy of current cultural norms for translating concepts and words into one another even for commonly used concepts. The last one is a successful compression from sense data to the fact that a common joke pattern is happening and inference that one is in a joke. I wish that we had a "Less Wrong Community" blog for off-topic but fun comments like the above to be top level posts, as well as an "instrumental rationality" blog for "self help" subject matter.
1thomblake11yYes and yes.
5NancyLebovitz11yI believe that humor requires harmless surprise, Harmlessness and surprise are both highly contextual, so what people find funny can vary quite a bit. One category of humor (or possibly an element for building humor) is things which are obviously members of a class, but which are very far from the prototype. Thus, an ostrich is funny while a robin isn't. This may not apply if you live in ostrich country-- see above about context.
3bgrah44911yI have spent a great deal of time thinking about humor, and I've arrived at a place somewhat close to yours. Humor is how we pass on lessons about status and fitness, and we do that using pattern recognition. I heard a comedian describe comedy by saying, "It's always funny when someone falls down. The question is, is it still funny if you push them?" He said for a smaller group of the population, it is. Every joke has a person being displayed as not fit - even if we have to take an object, or an abstraction, and anthropomorphize it. This is the butt of the joke. The more butts of a joke there are, the funnier the joke is - i.e., a single butt will not be that funny, but if there are several butts of a joke, or if a single person is the butt of several layers of the joke, it will be seen as funnier. The most common form of this is when the goals of the butt of a joke is divorced from their results. Joke 1: This is funny because Jeremy displays a lack of fitness by not being able to properly process the phrase "on TV." This has one butt - Jeremy. Joke 2: This joke has two butts. One is the muffin, which is being declared unfit for being bald. The other is the comedian's character, who is being displayed as needlessly paranoid toward a benign object (a muffin). Joke 3: This joke isn't that funny when displayed in text form - the comedy is in the performances, where both conversation participants are butts of the joke for arguing so intensely over something so petty. Joke 4: The butt of this joke is the traditional joke it's mocking. As for your outsiders' behavior: New student asks for both cream and lemon: Displays he is unfit by not understanding the purpose of what he's asking for. New employee swears and makes racist comments: This isn't funny in person, but it is funny if a few conditions are met. The first condition is that you're sufficiently removed from it (i.e., watching it on TV): Imminent threats aren't funny because this isn't a status lesson
3Nubulous11ySlight variant: Humour is a form of teaching, in which interesting errors are pointed out. It doesn't need to involve an outsider, and there's no particular class of error, other than that the participants should find the error important. If the guy sitting behind you starts moaning and grunting, if it's a mistake (e.g. he's watching porn on his screen and has forgotten he's not alone) then it's funny, whereas if it's not a mistake, and there's something wrong with him, then it isn't. Humour as teaching may explain why a joke isn't funny twice - you can only learn a thing once. Evolutionarily, it may have started as some kind of warning, that a person was making a dangerous mistake, and then getting generalised.
2NancyLebovitz11y [] New findings in the neurology of humor, and who finds what sort of jokes funny
2SilasBarta11yI'm glad you bring up this topic. I think that explanation makes a lot of sense: behavior that is wrong, but wrong in subtle ways, is good for you to notice -- you I.D. outsiders -- and so you benefit from having a good feeling when you notice it. Further, laughter is contagious, so it propagates to others, reinforcing that benefit. I want to present my theory now for comparison: A joke is funny when it finds a situation that has (at least) two valid "decodings", or perhaps two valid "relevant aspects". The reason it's advantageous in selection is that, it's good for you to identify as many heuristics as possible that fit a particular problem. That is, if you know what to do when you see AB, and you know what to do when you see BC, it would help if you remember both rules when you see ABC. (ABC "decodes" as "situation where you do AB-things" and as "situation where you do BC-things). Therefore, people who enjoy the feeling of seeking out and identifying these heuristics are at an advantage. To apply it to your examples: 1) It requires you to access your heuristics for "displayed on a TV screen" and "on top of a TV set". 2) It requires you to access your heuristics for "muffin as food" and "deficiencies of foods", not to mention the applicability of the concept of "baldness" to food. 3) Recognizing different heuristics for interpreting a date specification. 4) I don't know if this is a traditional joke: it became a traditional joke after the tradition of minister/priest/rabbi jokes. But anyway, its humor relies on recognizing that someone else can be using your own heuristics "minister/priest/rabbi = common form of joke", itself a heuristic. Food for thought...
3[anonymous]11ySir, I wish you no offense, but I happen to find my own theory more pleasing to the ear, so it befits me to believe mine rather than yours. And for some sentences that don't imitate someone behaving wrongly: I'd say that for the first three jokes, your theory works about as well as mine. Possibly worse, but maybe that's just my pro-me bias. The last one again doesn't fit the pattern. Recognizing that someone else can be using your own heuristics is not a type of being forced to interpret one thing in two different ways--is it? I notice that in the first three jokes, of the two interpretations, one of them is proscribed: "on TV" as "atop a television", a muffin as a non-cupcake, "next Wednesday" as the Wednesday of next week. In each case, the other interpretation is affirmed. Giving both an affirmed interpretation and a proscribed interpretation seems to violate the spirit of your theory. And a false positive comes to mind: why isn't the Necker cube inherently funny?
2gwern11yI too have a proto-theory. My theory is that humor is when there is a connection between the joke & punchline which is obvious to the person in retrospect, but not initially. Hence, a pun is funny because the connection is unpredictable in advance, but clear in retrospect; Eliezer's joke about the motorist and the asylum inmate is funny because we were predicting some other response other than the logical one; similarly for 'why did the duck cross the road? to get to the other side' is not funny to someone who has never heard any of the road jokes, but to someone who has and is thinking of zany explanations, the reversion to normality is unpredicted. Your theory doesn't work with absurdist humor. There isn't initially 1 valid decoding, much less 2.
5Alicorn11yI love absurdist humor. How many surrealists does it take to change a lightbulb? Two. One to hold the giraffe, and one to put the clocks in the bathtub.
5Jack11y"That isn't how the joke [] goes", said the cowboy [] hunched over in the corner of the saloon. The saloon was rundown, but lively. A piano played a jangly tune and the chorus was belted by a dozen drunken cattle runners, gold rushers, and ne're-do-wells. The whiskey flowed. In the distance, a wolf howled at the moon as if to ask it "Please, let the night go on forever." But over the horizon the sun objected like a stubborn bureaucrat. The bureaucrat slowly crossed the room, lighting everything at his feet as he moved. "Thank God I remembered to replace the batteries in this flashlight", the bureaucrat thought. The light bulb in his office had gone out again and would need to be replaced. Unfortunately that required a visit to the Supply Request Department downstairs. As he walked past the other offices he heard out of one "Fish!" as if the punchline to a joke had been given. But the bureaucrat heard no laughter. The Secretary of Supply Requests seemed friendly enough and she had even offered him something to drink. He took a swig and the continued: "the light bulb in my office has...". Gulp. "It needs to be replace..." The bureaucrat looked around. Suddenly he was feeling dizzy. Something was wrong. He looked down at the drink and then at the Secretary. She smirked. Her plan had succeeded. He had been poisoned! The bureaucrat didn't know what to do. He was terrified. He felt vertigo, as if he stood at the top of a tall ladder. The room started to spin. Counter-Clockwise. Then all of a sudden everything went black. A few seconds later he felt the room spinning again-- strangely, in the opposite direction-- and suddenly, he lit up.
3orthonormal11yHow many mathematicians does it take to screw in a lightbulb? One. They call in two surrealists, thus reducing it to an already solved problem. ?

What would it be like to be a rational atheist in the fifteenth century, and know beyond all hope of rescue that everyone you loved would be annihilated, one after another as you watched, unless you yourself died first? That is still the fate of humans today; the ongoing horror has not changed, for all that we have hope.

That may be a recent innovation; it wasn't threaded in the days when Eliezer's articles appeared there.

I made a couple posts in the past that I really hoped to get replies to, and yet not only did I get no replies, I got no karma in either direction. So I was hoping that someone would answer me, or at least explain the deafening silence.

This one isn't a question, but I'd like to know if there are holes in my reasoning.

Here, I had a question:

I looked at your consciousness comment. First, consciousness is notoriously difficult to write about in a way that readers find both profound and comprehensible. So you shouldn't take it too badly that your comment didn't catch fire.

Speaking for myself, I didn't find your comment profound (or I failed to comprehend that there was profundity there). You summarize your thesis by writing "Basically, a qualium is what the algorithm feels like from the inside for a self-aware machine." (The singular of "qualia" is "quale", not "qualium", btw.)

The problem is that this is more like a definition of "quale" than an explanation. People find qualia mysterious when they ask themselves why some algorithms "feel like" anything from the inside. The intuition is that you have both

  1. the code — that is, an implementable description of the algorithm; and

  2. the quale — that is, what it feels like to be an implementation of the algorithm.

But the quale doesn't seem to be anywhere in the code, so where does it come from? And, if the quale is not in the code, then why does the code give rise to that quale, rather than to some other one?

T... (read more)

I understand there is a distinction. Would you agree that RSI systems are conceptually a subset of self-modifying (SM) systems? One that we don't understand what exact properties make a SM system one that will RSI. Could you theoretically say why EURISKO didn't RSI?

I was interested in how big a subset. The bigger it is the more dangerous, the more easily we will find it.

2gwern11ySure. In fact, some of the Lenat quotes on LW even tell you why. As a hack to defeat 'parasitic' heuristics, Lent (& co.?) put into Eurisko a 'protected kernel' which couldn't be modified. This core was not good enough to get everything going, dooming Eurisko from a seed AI perspective, and the heuristics never got anywhere near the point they could bypass the kernel. Eurisko was inherently self-limited.

Well, that's where I got it.

Assuming the general social norms for spoilers thing for "spoilability", not whether it actually ruins entertainment for those who don't know the story yet or not.

Convincing Dawkins would be a great strategy for promoting cryonics... who else should the community focus on convincing?

Friends and family. They are the ones I care about most. (And, most likely, those that others in the community care about most too. At least the friends part. Family is less certain but more significant.)

2MichaelBishop9ySure, convince those you love. I was asking who you should try to convince if your goal is convincing someone who will themselves convince a lot of other people.

I found this (scroll down for the majority of articles) graph of all links between Eliezer's articles a while ago, it could be be helpful. And its generally interesting to see all the interrelations.

Yes- it's very natural for the ongoing community progression of LW, but not great for archiving; we're pulling up the latter after we've climbed it.

Yes, yes, and the same is true of pet adoption! A friend of mine found this ultra-cute little kitten, barely larger than a soda can (no joke). I couldn't help but adopt him and take him to a vet, and care for that tiny tiny bundle of joy, so curious about the world, and so needing of my help. I named him Neko.

So there, we have another contravention of the gene's wishes: it's a pure genetic cost for me, and a pure genetic benefit for Neko.

Well, I mean, until I had him neutered.

You're really not grasping the concept of biasproofing, are you? "A confederate" taints the experiment.

Not performing the experiment taints it even more. But perhaps by now we've learned enough of each other's claims. I leave it to you to find a way forward.

I refer you in that case to the canonical example...

2arundelo11yRoger Ebert responding to a reader about [edit: the "spoiler" in the title of] The Assassination of Jesse James by the Coward Robert Ford [] . (Warning: contains a Sixth Sense spoiler.)

This is pretty self-important of me but I'd just like to warn people here that someone is posting at OB under "Jack" that isn't me so if anyone is forming a negative opinion of me on the basis of those comments- don't! Future OB comments will be under the name Jack (LW). The recent string of comments about METI are mine though.

This is what I get for choosing such a common name for my handle.

Apologies to those who have read this whole comment and don't care.

An inquiry regarding my posting frequency:

While I'm at the SIAI house, I'm trying to orient towards the local priorities so as to be useful. Among the priorities is building community via Less Wrong, specifically by writing posts. Historically, the limiting factor on how much I post has been a desire not to flood the place - if I started posting as fast as I can write up my ideas, I'd get three or four posts out a week with (I think) no discernible decrease in quality. I have the following questions about this course of action:

  1. Will it annoy people? B

... (read more)
7byrnema11yAs your goal is to build community, I would time new posts based on posting and commenting activity. For example, whenever there is a lull, this would be an excellent time to make a new post. (I noticed over the weekend there were some times when 45 minutes would pass between subsequent comments and wished for a new post to jazz things up.) On the other hand, if there are several new posts already, then it would be nice to wait until their activity has waned a bit. I think that it is optimal to have 1 or 2 posts 'going on' at a time. I prefer the second post when one of them is technical and/or of focused interest to a smaller subset of Less Wrongers. (But otherwise no limit on the rate of posts.)
5Eliezer Yudkowsky11yI'd say damn the torpedoes, full speed ahead. If people are annoyed, let them downvote. If posts start getting downvoted, slow down. Your posts have generally been voted up. If now is the golden moment of time where you can get everything said, then for the love of Cthulhu, say it now!
1Alicorn11yI don't anticipate being so obnoxiously prolific that people collectively start voting my posts negative such that they stay that way. But people already sometimes register individual downvotes on posts that I make, and I don't want that to happen on a larger fraction of posts due to increased frequency, because I can't reliably distinguish between "you must have had an off day, this post is not up to scratch" and "please, please, please shut up".
2wedrifid11yPost away. The best signal to anticipate from the audience in this case is "how many votes in total do I expect if I post at full speed vs how many votes I expect if I posted less frequently and so ended up writing less posts overall". Increased frequency may give you less votes per post. Frequent posts from the same author may be less desired and if you post less you may only be giving the best posts. But if the net expectation is higher for more prolific posting then that can be interpreted as "the community would prefer you to post faster than a spambot". Even if you expected a total of less karma for more posts I wouldn't say that means you ought not post more. So long as your posts are still breaking the 10 mark we clearly don't mind your contribution. There are probably other benefits to you from posting than maximising the benefit to lesswrong. I find writing helps clarify my thinking for example. So as long as you are still being received somewhat positively you are free to type away.
1Paul Crowley11yPost as much as you like, if you think it's good quality; I promise to say if I start to think slowing down would be a good idea.
1Eliezer Yudkowsky11yI don't mean "downvoted negative" just "downvoted relative to other posters".
2PeerInfinity11yone obvious idea that I didn't notice anyone else mention: Another option is to go ahead and write the posts as fast as you think is optimal, but if you think this is too fast to actually post the stuff you've written, then you can wait a few days after you wrote it before posting. LW has a handy "drafts" feature that you can use for that. This also has the advantage that you have more time to improve the article before you post it, but the disadvantage that you may be tempted to spend too much time making minor, unimportant improvements. Another disadvantage is that feedback gets delayed.
1Alicorn11yIf I sit on posts for too long, I start second-guessing myself and often wind up deleting them.
1Alicorn11yA related question: If I have a large topic to cover, should I cover it in one post, or split it up along convenient cleavage planes and make it a sequence? (If I make sequences, I think I'll learn my lesson from the last one I tried and write it all before posting anything, so I don't post 2/3 of it and then stop.)
3Paul Crowley11yI really like the "sequences" approach - it's easier to read and digest a chunk at a time, and it focusses discussion well, too.
3RobinZ11yLong posts are more offputting than short ones, and individual steps are more likely to be correct than entire theorems - both of these points would suggest posting sequences preferentially. As for a specific reference on length: thirty-three hundred words sharply focused on a single, vivid subject [] is pushing the upper limit of what I find comfortable to attack in a single sitting.
1Eliezer Yudkowsky11yPosting 2/3 of a sequence and stopping is fine if people turn out not to be interested. I recommend fast posting and fast feedback.

Yeah, so... I'm betting if we could hook this guy up to a perfect lie detector, it would turn out to be a conscious scam. Or am I still underestimating human insanity by that much?

3thomblake11yYes. People dismiss the scientific evidence weighing similarly against them on many issues in the news every day. There's nothing spectacular about finding someone who does it regarding the Earth being flat, especially given that an entire society has existed for hundreds of years to promote the idea.
1MichaelHoward11yThat you see this as a particularly extreme case of insanity (even in an apparently intelligent, lucid, fully-functioning person) is far more shocking to me than this guy. Maybe I've just seen too many Louis Theroux documentaries.

Anyone who can travel through time can mount a pretty impressive apocalypse and announce whatever it is about the nature of reality he cares to. He might even be telling the truth.

I don't understand why true randomness is a problem. Is there something so wrong with probabilistic determinism?

Yeah. I just updated it again. I didn't realize anyone was actually looking at it... :P

Recently I burned out on the sequences and am taking a break to gorge myself on other subjects. I tend to absorb large scale topics in rotation. It helps me stay focused over longer distances and has an added benefit of making me reread stuff that didn't stick the first time through. The weekly study group will also help data retention.

Other data points that may be relevant: I have participated in a lot of online discussions; I have moved cross country into a drastically... (read more)

It's tricky. As Socrates noted in the Apology, it's much easier to convince someone in a one-on-one conversation than to have a general argument to convince anyone in general.

White people have not unanimously decided to do what is necessary to end the ongoing oppression of non-white people, let alone erase the effects of past oppression.

Edit: Folks, I am not accusing you or your personal friends of anything. I have never met most of you. I have certainly not met most of your personal friends. if you do not agree with the above comment, please explain why you think there is no longer such a thing as modern-day racism in white people.

This seems to suggest that it is easier to tell the difference between Coke and Pepsi than it is to tell the difference between regular and diet. I can tell the difference between all of them, but the first is a lot harder and I think that experience is pretty common. Most diet drinks use aspartame as a sugar substitute and aspartame leaves a very distinctive aftertaste in my mouth.

I've always thought Clippy was just a funny inside joke -- thought unfortunately not always optimally funny. (Lose the Microsoft stuff, and stick to ethical subtleties and hints about scrap metal.)

I was mostly referring to how the reasoning had to deal a gradually accreting set of rules, each one constructed in the service of narrative (that is, fun) instead of being a realistic constraint. I really did mean Calvinball.

Let me build on this. You say (and I agree) that fixing the damage caused by vitrification is much harder than fixing most causes of death. Thus, by the time that devitrification is possible, very few new people will be vitrified (only people who want a one-way trip to the future).

This leads me to 2 conclusions: 1) Most revivals will be of people who were frozen prior to the invention of the revivification technology. Therefore, if anyone is revived, it is because people want to revive people from the past. 2) The supply of people frozen with a given te... (read more)

Even setting aside a post-FAI economy, why should this be the case? Your PS3 metaphor is not applicable. Owners of old playstations are not an unserved market in the same way that older frozen bodies are. If PS(N) games are significantly more expensive than PS(N+1) games, people will simply buy a PS(N+1). Not an option for frozen people; older bodies will be an under served market in a way PS3 owners cannot be.

If there's a "mass market" for revivals, clearly people are getting paid for the revivals, somehow. I see no reason why new bodies w... (read more)

You're right - bad metaphor. What about the vision one?

The people who are running the LHC aren't doing basic research?

In order for cryonics to work in principle, they will already have to know how to fix whatever you died of, whether it be cancer or a heart attack or being crushed in a car accident. How much more difficult do you estimate it would be to fix the problem of vitrification (fixing vitrification damage and preventing de-vitrfication damage) than fix the cause of your death?

First, I estimate it will be orders of magnitude harder to fix that damage than to fix the cause of death. Fixing cancer or many other diseases would likely be a matter of intervening wi... (read more)

Oh, look honey: more proof wine tasting is a crock:

A French court has convicted 12 local winemakers of passing off cheap merlot and shiraz as more expensive pinot noir and selling it to undiscerning Americans, including E&J Gallo, one of the United States' top wineries.

Cue the folks claiming they can really tell the difference...

6Morendil11yThere's plenty of hard evidence that people are vulnerable to priming effects and other biases when tasting wine. There's also plenty of hard evidence that people can tell the difference between wine A and wine B, under controlled (blinded) conditions. Note that "tell the difference" isn't the same as "identify which would be preferred by experts". So, while the link is factually interesting, and evidence that some large-scale deception is going on, aided by such priming effects as label, marketing campaigns and popular movies can have, it seems a stretch to call it "proof" that people in general can't tell wine A from wine B. Rather, this strikes me as a combination of trolling and boo lights: cheaply testing who appears to be "on your side" in a pet controversy. How well do you expect that to work out for you, in the sense of "reliably entangling your beliefs with reality"?
1SilasBarta11yI think I'm entangling my beliefs with reality very well, by virtue of extracting all available information from phenomena rather than retreat to evidence that agrees with me. (Let's not forget, I didn't start out thinking that it was all BS.) For example, did you stop to notice the implications of this: How does that compare to the priming effects for other drinks? Does it matter? But what would be the appropriate comparison? They were passing of as expensive, something that's actually cheap. Where else would that work so easily, for so long? Normally, if you tried that, it would be noticed quickly, if not immediately, by virtually everyone. What if you tried to pass off 16 oz of milk as 128? Or spoiled milk as milk expiring in a week? Then, factor in how much difference is claimed to exist in wine vs. milks. Who's optimally using evidence here?
5CronoDAS11yArt forgeries. (Which shows that the value of the painting is determined by the status of the artist and not the quality of the art.) If I can paint a painting that convinces experts that it was painted by [insert expert painter here], does that mean I'm as good an artist as said painter? (Assuming that my painting isn't a literal copy of someone else's.)
3SilasBarta11yWhich, like wine, is another example of a path-dependent collective delusion that's not Truly Part of [] our values. (That is, our valuation of the work wouldn't survive deletion of the history that led to such a valuation.) Very nearly yes, it does, modulo a few factors. If you produced it after the artist, then you are benefiting from the artist's already having identified a region of conceptspace that you did not find yourself. (If the art is revered because of the artist's social status, that it wasn't even much of an accomplishment to begin with.) To put it another way, you produced the work after "supervised learning", while the artist didn't need that particular training. If you can pass off a previous work of yours as being one of the artist's, that definitely makes you better.
1komponisto11yWho is "we", here? The problem I have is not that you're wrong, for the people you're talking about; it's that you (probably) overestimate the size and/or importance of that population. You're not telling the whole truth, in effect. There are plenty of people who like paintings for the way they look, and would happily buy the work of a lesser-known artist at a cheap price if they liked it. Yes, some people use art to status-signal, but some people also actually like art. (There may even be a nonempty intersection!)
4Morendil11yYou seem to want a contest. The other option, where we are both "on the side of truth", appeals to me more. We're fortunate in having different experiences in the domain of taste. I'm one of those people who like wine, and I'm confident I can identify some of its taste characteristics in blind tests. So, predictably I resent language which implies I'm an idiot, but I'm open to inquiry. Our investigation should probably begin "at the crime scene", that is, close to what evidence we can gather about the sense of taste. So, yes, we could examine similar priming effects on other drinks. I have a candidate in mind, but what I'd like to ask you first is, suppose I name the drink I have in mind and we then go look for evidence of fraud in its commerce. What would it count as evidence of if we found no fraud? If we did find it? Which one would you say counts as evidence that "people can't tell the difference" between wines?
4Cyan11yYou can easily test yourself if you have a confederate. I recommend a triangle test [].
2jpet11yIf "top winery" means "largest winery", as it does in this story, I don't see how it says anything about the ability of tasters to tell the difference. Those who made such claims probably weren't drinking Gallo in the first place. I think it's closer to say they were passing off as cheap, something that's actually even cheaper. Switch the food item and see if your criticism holds: Wonderbread, America's top bread maker, was conned into selling inferior bread. So-called "gourmets" never noticed the difference! Bread tasting is a crock.
1SilasBarta11yIf people made such a huge deal about the nuances in the taste of bread, while it also "happened" to have psychoactive effects that, gosh, always have to be present for the bread to be "good enough" for them, and cheap breads were still normally several times the cost of comparable-nutrition food, then yes, the cases would be parallel. (Before anyone says it: Yes, I know bread as trace quantities of alcohol, we're all proud of what you learned in chemistry.)

Seaching for some corroboration of this I came across this little gem in the Simple English Wikiquotes:

Replying to following question by Deborah Solomon in Questions for Douglas Hofstadter: "Your entry in Wikipedia says that your work has inspired many students to begin careers in computing and artificial intelligence." He replied "I have no interest in computers. The entry is filled with inaccuracies, and it kind of depresses me." When asked why he didn't fix it, he replied, "The next day someone will fix it back."[6]


... (read more)

No reply. Just so you know, the collective position on testing here is bizarre.

How you can think that superintelligent agents are often dangerous AND that a good way of dealing with this is to release an untested one on the world is really quite puzzling.

Hardly anyone ever addresses the issue. When they do, it is by pointing to AI box experiments, which purport to show that a superintelligence can defeat a lesser intelligence, even if well strapped down.

That seems irrelevant to me. To build a jail, for the smartest agent in the world, you do not use vas... (read more)

How do I turn this off?

2Clippy11yAre you sure you want to turn this feature off?

I do not understand what general category of malfunction corresponds to this particular mistake.

Seems to me it's implicitly an example of the common category "assuming that charity (or FAI) will not be an important factor in revival".

You've been in the transhumanist community for, what, at least 10 years?

20, my young friend.

I honestly have no clue what could possibly be going wrong in your mind at this moment. I do not understand what general category of malfunction corresponds to this particular mistake.

And that should suggest to you that the mistake may be yours.

2Paul Crowley11yIt's certainly evidence in favour of that position, but not enough...

Would that all dicks were so amusing.

I would think post-Singularity historians would like nothing better than to revive the earliest people they could.

The real question is whether it's a tinny word or a woody word.

You ran into a Poe's-Law type problem: your joke was indistinguishable from stupidity.

Heilmeier's Catechism, a set of questions credited to George H. Heilmeier that anyone proposing a research project or product development effort should be able to answer.

3Eliezer Yudkowsky11y"How much will it cost?" "How long will it take?" Who the hell is supposed to be able to answer that on a basic research problem?
5PhilGoetz11yAnyone applying for grant money. Anyone working within either the academic research community or the industrial research community or the government research community. Gentleman scientists working on their own time and money in their ancestral manors are still free to do basic research.
2RichardKennaway11yNowadays, everyone who applies for a grant.
2Morendil11yYou can take them as a calibration exercise. "I don't know" or "Between a week and five centuries" are answers, and the point of asking the question is that some due diligence is likely to yield a better (more discriminating) answer. Someone who had to pick one of two "basic research problems" to fund, under constraints of finite resources, would need estimates. They can also provide some guidance to answer "How long do we stick with this before going to Plan B?"
2whpearson11yInteresting, but some of the questions aren't easy to answer. For example if you were asking the question to someone involved in early contraception development, do you think they could of predicted what demographic/birth rate changes it would have? Similarly someone inventing a better general machine learning technique, (useful for surveillance to robot butlers) could they enumerate the variety of ways it would change the world? For AI projects, even weak ones, I would ask how they planned to avoid the puppet problem [].
4Morendil11yThe point of such "catechisms" isn't so much to have all the answers, rather to ensure that you have divided your attention evenly among a reasonable set of questions at the outset, in an effort to avoid "motivated cognition" - focusing on the thinking you find easy or pleasant to do, as opposed to the thinking that's necessary. The idea is to improve at predicting your predictable failures []. If this kind of thinking turns up a thorny question you don't know how to answer, you can lay the current project aside until you have solved the thorny question, as a matter of prudent dependency management. A related example is surgery checklists. They work (see Atul Gawande's Better). Surgeons hate them - their motivated cognition focuses on the technically juicy bits of surgery, they feel that trivia such as checking which side limb they're operating on is beneath them.
2whpearson11yI'm a big believer in surgery checklists. However I'm yet to be convinced that the catechisms will be beneficial unaltered to any research project. A lot of science is about doing experiments that we don't know the outcomes of and serendipitously discover things. Two examples that spring to mind are superconductivity [] and fullerene production []. If you asked each of the discoverers to justify their research by the catechisms you probably would have got no where near the actual results. This potential for serendipity should be built into the catechisms in some way. That is the answer "For Science!" has to hold some weight, even if it is less weight than is currently ascribed to it.

I know about the experiment you mention, and it partly motivated my suggestion; I just subjectively find "yellowness" and "blueness" more qualious than "upness" or "leftness".

In my experiment, "yes" would mean that there would be no dissonance between memories and perceptions, that I would just not feel that the trees are red or purple, but green, and find the world "normal". That I would, one day, cease to feel the need to get rid of the color-changing glasses, and my aesthetic preferences would ... (read more)

I wasn't think of evildoers. I was thinking of people who are just different, and have their own culture, traditions and way of life.

2Roko11yI think that thinking in terms of good and evil belies a closet-realist approach to the problem. In reality, there are different people, with different cultures and biologically determined drives. These cultural and biological factors determine (approximately) a set of traditions, worldviews, ethical principles and moral rules, which can undergo a process of reflective equilibrium to determine a set of consistent preferences over the physical world. We don't know how the reflective equilibrium thing will go, but we know that it could depend upon the set of traditions, ethical principles and moral rules that go into it. If someone is an illiterate devout pentecostal Christian who lives in a village in Angola, the eventual output of the preference formation process applied to them might be very different than if it were applied to the typical LW reader. They're not evil. They just might have a very different "should function" than me.
6steven046111yI think part of the point of what you call "moral anti-realism" is that it frees up words like "evil" so that they can refer to people who have particular kinds of "should function", since there's nothing cosmic that the word could be busy referring to instead. If I had to offer a demonology, I guess I might loosely divide evil minds into: 1) those capable of serious moral reflection but avoiding it, e.g. because they're busy wallowing in negative other-directed emotion, 2) those engaging in serious moral reflection but making cognitive mistakes in doing so, 3) those whose moral reflection genuinely outputs behavior that strongly conflicts with (the extension of) one's own values. I think 1 comes closest to what's traditionally meant by "evil", with 2 being more "misguided" and 3 being more "Lovecraftian". As I understand it, CEV is problematic if most people are "Lovecraftian" but less so if they're merely "evil" or "misguided", and I think you may in general be too quick to assume Lovecraftianity. (ETA: one main reason why I think this is that I don't see many people actually retaining values associated with wrong belief systems when they abandon those belief systems; do you know of many atheists who think atheists or even Christians should burn in hell?)
2Unknowns11y"One main reason why I think this is that I don't see many people actually retaining values associated with wrong belief systems when they abandon those belief systems; do you know of many atheists who think atheists or even Christians should burn in hell?)" One main reason why you don't see that happening is that the set of beliefs that you consider "right beliefs" is politically influenced, i.e. human beliefs come in certain patterns which are not connected in themselves, but are connected by the custom that people who hold one of the beliefs usually hold the others. For example, I knew a woman (an agnostic) who favored animal rights, and some group on this basis sent her literature asking for her help with pro-abortion activities, namely because this is a typical pattern: People favoring animal rights are more likely to be pro-abortion. But she responded, "Just because I'm against torturing animals doesn't mean I'm in favor of killing babies," evidently quite a logical response, but not in accordance with the usual pattern. In other words, your own values are partly determined by political patterns, and if they weren't (which they wouldn't be under CEV) you might well see people retaining values you dislike when they extrapolate.
3wedrifid11yAlternately: They're evil. They have a very different 'should function' to me.

Endless YouTube nutshot videos, Anonymous hacking#Epilepsy_Foundation_forum_invasion) an epilepsy support forum with flashing GIFs, the infamous banana peel on the sidewalk... Not particularly high-brow humour but many people find such things amusing.

5Cyan11y"Tragedy is when I cut my finger. Comedy is when you fall into an open sewer and die." - Mel Brooks

Well, what if he were sent back to Ancient Greece (and magically acquired the ability to speak Greek)? Even if he got all the math perfectly right, who would care? Or even understand it?

He would then spend the rest of his life ensuring that it is preserved. If necessary he would go around hunting for obscure caves with a chisel in hand. Depending, of course, on how much he cares about influencing the future of the universe as opposed to other less abstract goals.

Intrinsically they aren't optimization processes but they seem computationally expressive enough for an optimization process to be implemented on them

But they aren't optimization processes. It doesn't matter if they could implement one, they don't. You might as well point to any X86 chip and ask why it doesn't RSI.

I don't understand your explanation. You are apparently saying that quale (you seem to deliberately misspell the word, why?) is how the algorithm feels from inside. Well, I agree, but in the same time I think that "quale" is only a philosopher's noble word for "feel from inside". The explanation looks like a truism.

I have always been (and still am) confused by questions like: How other people perceive colors? Do they feel it the same way as I do? Are there people who see the colors inverted, having the equivalent of my feeling of "... (read more)

2Steve_Rayhawk11yI once heard of people at UCSD who had plans to experiment with inverted spectrum goggles. Jonathan Cohen [] would know more.

Most technological developments seem to go from "We don't know how to do this at all" to "We know how to do this, but actually doing it costs a fortune" to "We know how to do this at an affordable price." WBE could be an exception, though, and completely skip over the second stage.

I am curious: are you very old or suffering from a fatal disease? I am 25 and healthy, so "lifetime" probably means something different to me...

I mentioned the AI-talking-its-way-out-of-the-sandbox problem to a friend, and he said the solution was to only let people who didn't have the authorization to let the AI out talk with it.

I find this intriguing, but I'm not sure it's sound. The intriguing part is that I hadn't thought in terms of a large enough organization to have those sorts of levels of security.

On the other hand, wouldn't the people who developed the AI be the ones who'd most want to talk with it, and learn the most from the conversation?

Temporarily not letting them have the power to g... (read more)

5Paul Crowley11yI am by and large convinced by the arguments that a UFAI is incredibly dangerous and no precautions of this sort would really suffice. However, once a candidate FAI is built and we're satisfied we've done everything we can to minimize the chances of unFriendliness, we would almost certainly use precautions like these when it's first switched on to mitigate the risk arising from a mistake.
2xamdam11yThis might be stupid (I am pretty new to the site and this possibly has come up before), I had a related thought. Assuming boxing is possible, here is a recipe for producing an FAI: Step 1: Box an AGI Step 2: Tell it to produce a provable FAI (with the proof) if it wants to be unboxed. It will be allowed to carve of a part of universe to itself in the bargain. Step 3: Examine FAI the best you can. Step 4: Pray
5Nick_Tarleton11ySomething roughly like this was tried in one of the AI-box experiments []. (It failed.) []

The idea of universal intelligence is not a bug, it is a feature. It is mainly due to Legg/Hutter that we have that concept in the first place - and it is a fine one.

I think I need to clarify here.

I am personally convinced (am a one-time donor myself), but the optimal charity argument in favour of Friendly AI research and development (which will be fully developed in this paper) is something I can use with my friends. They are pretty much the practical type and will definitely respond to wanting more bang for their buck and where their marginal rupee of charity should go.

There are inferential gaps. And when me, a known sci-fi fan presents the argument, I get all sorts of looks. If I have a peer reviewed paper to show them, that would work nicely in my favour.

Not necessarily. If a human brain with no software tricks requires 10^20 CPS (a very high estimate), then (according to Kurzweil, take with grain of salt) the computational capacity will be there by ~2040. However, it's certainly possible that we don't get the software until 2050, at which point anyone with a couple hundred dollars can run one.

Would you consider SENS a viable alternative to SIAI? Or do you think ending aging is also impossible/something to be put off?

Yes, who today cares what any Greek mathematician had to say...

Now you're just moving the goal posts.

1Nisan11ySomeone doesn't like Bach because he was traumatized by an exposure to classical music at a tender age? Give me a break. Music is like languages, not math -- the surest way to learn to like Bach is full immersion at a young age, not a graduated curriculum that starts from "lower" forms of music.
  1. 80
  2. 20
  3. Pretty small.

Sure. But if GiveWell isn't going to do it then someone should. Are their budgets public? So many people here are skeptical of regular charities what evidence is there that these charities are different?

2Kevin11yI don't think they publish a full budget but there is a breakdown of what the current fundraising drive is for. []

If the character isn't deliberately made confused (as opposed to paperclip-preferring, for example), resolving character's confusion presumably helps the author as well, and of course the like-confused onlookers.

Check with here, for example, though:

The "AI researcher" definitions in particular seem to be much the same as the definition of instrumental rationality.

I'd guess CI + SA > Alcor > CI.

7Kevin11yI didn't know you thought CI + SA was actually better than Alcor regardless of cost. Have you said that in more words elsewhere on this site?

Take a look at who's posting it.

Er, yes, that's exactly why I wasn't sure.

I've been wondering what the existance of Gene Networks tells us about recursively self improving systems. Edit: Not that self-modifying gene networks are RSIS, but the question is "Why aren't they?" In the same way that failed attempts at flying machines tell us something, but not much, about what flying machines are not. End Edit

They are the equivalent of logic gates and have the potential for self-modification and reflection, what with DNAs ability to make enzymes that chop itself up and do so selectively.

So you can possibly use them as evide... (read more)

3[anonymous]11yIt seems like in gene networks, every logic gate has to evolve separately, and those restriction enzymes you mention barely do anything but destroy foreign DNA. That's less self-modification potential than the human brain.
2whpearson11yThe inability to create new logic gates is what I meant by the systems having low memory. In this case low memory to store programs. Restriction enzymes also have a role in the insertion of plasmids [] into genes. An interesting question is: If I told you about computer model of evolution with things like plasmids, controlled mutation [\(D\] J_recombination); would you expect it to be potentially dangerous? I'm asking this to try to improve our thinking about what is and isn't dangerous. To try and improve upon the kneejerk "everything we don't understand is dangerous" opinion that you have seen.

Can't watch video from here, and in any case given the much greater investment of time required I'd want to know more about it to start watching. Anyone who's seen it care to say if there are any new or good arguments in there?

3Furcas11yIt's the usual 'Rapture of the Nerds' spiel.
2Paul Crowley11yThanks to you and whpearson for taking the time to find out so the rest of us don't have to. Voted timtyler down for wasting your and our time. Edit: removed downvote, see below.
3timtyler11yEr, this is pretty relevant and on-topic material, IMHO! Jaron Lanier is a fruitcake - but I figure most participants here already knew that. You may not personally be interested in what famous geek critics have to say about "the Tech World's New Religion" - but it seems bad to assume that everyone here is like that.
5Paul Crowley11yHmm, I didn't see it that way. Removed downvote. But videos are a pain; you could do us a favour next time by saying a few more words about whether you're recommending it or just posting FYI. Otherwise there's a Gricean implication that you judge it worth our time, I think.
8SilasBarta11yThere should be a policy, or strong norm, of "No summary, no link" when starting a thread with a suggested link. That summary should tell the key insights gained, and what about it you found unique. I hate having to read a long article -- or worse, listen to a long recording -- and find out it's not much different from what I've heard a thousand times before. (That happens more than I would expect here.) Of course, you shouldn't withhold a link just because Silas (or anyone else) already read something similar ... but it tremendously helps to know in advance that it is something similar.
2timtyler11yYes, I don't like "teaser" links much either. I did give the author, title and a three word synopsis - but more would probably have helped. On the other hand, I didn't want to prejudice watchers too much by giving my own opinion up front.
4gwern11yThey may not; it takes a while for one to figure out that he really is a fruitcake - individual columns or essays tend to sound like perfectly respectable contrarianism. I first began reading his articles in Discover and it wasn't until his 'Digital Maoism' essay that "he really is a nut!" occurred to me.
2whpearson11yListening to the longer version isn't so bad. The snippet was definitely the most objectionable. It appears that Lanier thinks AI is suffering from the puppet problem bought on by taking the Turing test too seriously. The puppet problem is that computers can be used to implement puppets. Things that fake being intelligence. Imagine Omega makes a program for the Turing Test that looks intelligent by predicting you and having the program output intelligent sounding responses at different times, so that you (and only you!) think it is intelligent but you are really talking to the advanced equivalent of an answer phone*. So he thinks that AIs are going to be puppets. Which is a semi-reasonable opinion to come to if you just look at chatbots. However Lanier doesn't, but should, argue that computers can only be puppets. Edited: For clarity. *I think Eliezer said something like if you see intelligent behaviour you should guess that there is an intelligence somewhere, it may just not be in the system that appears intelligent. I'm not organised enough to keep a quote file. Anyone?
4Zack_M_Davis11y"GAZP vs. GLUT []":
2whpearson11yIn the short version, mainly saying the singularity is a nutty concept and making strange comments about Turing. It does not encourage me to watch the longer version. I've found the audio [] for the longer version. Which I may listen to at some point.

It bugs me that people don't think of this one more often. It's basically an anime about how science affects the world and its practitioners.

(Disclaimer: Far too many convenient coincidences/idiot balls IIRC. It's a prime target for a rationalist rewrite.)

There is some previous discussion which suggests that not everyone here would be happy to see LW as a "rationality dojo".

The term "dojo" has favorable connotations for me, partly because one of my secret identity's modest claims to fame is as a co-originator of the "Coding Dojo", an attempt to bring a measure of sanity back to the IT industry's horrible pedagogy and hiring practices.

However these connotations might be biasing my thinking about whether using the "dojo" metaphor as a guide to direct the evolution of LW... (read more)

Once again, which people? You are not talking about the millions of people who go to fertility clinics, presumably. Those people apparently genuinely want to procreate.

Many drug addicts seem to share Cypher's perspective on this issue. They want the pleasure, and aren't too picky about where it comes from.


The child wants to eat the donut rather than store up calories or stimulate certain nerves. It still wants to eat the donut even if the sugar has been replaced with artificial sweetener.

People want sex rather than procreate or stimulate certain nerves. They still want sex even if contraception is used.

If there's a pun I'm afraid it's unintentional - are you referring to the literal meaning of "canon" in this context?

Did you pick that movie for that reason, or because that's what TV Tropes used? Because I've never seen it, but I do know that Macduff was not of woman born - and Macbeth is rather better known.

Edit: Better still is "Romeo and Juliet die at the end".

That's not clear to me, but I acknowledge that it doesn't affect the original question.

If someone just observed events in 2011 and planted a book describing them in 1200, the 2011 resulting from the history where the book existed would be different from the 2011 he observed.

2Paul Crowley11yDepends if it's type one time travel []. Fictional examples: Twelve Monkeys, The Hundred Light-Year Diary.
3thomblake11yI think the important bit here is that even if you could just "play time backwards" and watch again, there's no reason to think you'd end up in the same Everett branch the next time around.
1Document11yInsofar as I understand that page, that would mean that the world worked even less the way we thought it did.
1FAWS11yMakes perfect sense to me if you assume a single time-line. (This might be a big assumption, but probably less big than the truth of sufficiently strange prophecies.) You can think of this time line as having stabilized after a very long sequence of attempts at backward time travel under slightly different conditions. Any attempt at backward time travel that changes its initial conditions means a different or no attempt at time travel happens instead. Eventually you end with a time-line where all attempts at backward time travel exactly reproduce their initial conditions. We know that we live in that stabilized time-line because we exist (though the details of this timeline depend on how people who don't exist, but would have thought they exist for the same reasons we think we exist, would have acted, had they existed).
2FAWS11yBy the way, that sort of time-travel gives rise to Newcomb-like problems: Suppose you have access to a time-machine and want to cheat on a really important exam (or make a fortune on the stock marked or save the world or whatever. The cheating example is the simplest). You decide to send yourself at a particular time a list with the questions after taking the exam. If you don't find the list at the time you decided you know that somehow your attempt at sending the list failed (you changed your mind, the machine exploded in a spectacular fashion, you were caught attempting to send the list ...). But if you now change your mind and don't try to send the list there never was any possibility of receiving the list in the first place! The only way to get the list if for you to try to send the list even if you already know you will fail, so that's what you have to do if you really want to cheat. And if you really would do that, and only then, you will probably get the list at the specified time and never have to do it without knowing you succeed, but only if your pre-commitment is strong enough to even do it in the face of failure. And if you would send yourself back useful information at other times even without having either received the information yourself or pre-commited to sending that particular information you will probably receive that sort of information.
1FAWS11yWhy was this post voted back down to 0 after having been at 2? Newcomb-like problems are on-topic for this site and I would think having examples of such problems in a scenario not specifically constructed for them is a good thing? If it was because time travel is off topic wouldn't the more prudent thing have been voting down the parent? The same if the time travel mechanics are considered incoherent (though I'd be really interested in learning why?) . If you think this post doesn't actually describe anything Newcomb-like I would like to know why. Maybe I misunderstood the point of earlier examples here, or maybe I didn't explain things sufficiently? Or is it just that the post was written badly? I'm not really happy with it, but I don't see how I could have made it much clearer.
1wedrifid11yIt's an interesting point. It actually came up in the most recent Artemis Fowl novel, when he managed to 'precommit' himself out of a locked trunk in a car. :)

Respondents, please upvote thomblake's comment if this seems like an acceptable rule of thumb.

Edit: And likewise for other things people say if those seem like good ideas.

What makes this relevant to LW participants?

I just failed the Wason selection task. Does anyone know any other similarly devilish problems?

3Cyan11yHere's a classic: Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations. Which is more probable? 1. Linda is a bank teller. 2. Linda is a bank teller and is active in the feminist movement. Answer here [].

In proportion to the complexity of the claim given that humans exist, which is much lower (=> higher prior) than its complexity in a simple encoding, since Scientology is the sort of thing that a human would be likely to propose.

Following the same analogy, you can translate it as "if only the God did in fact exist, ...". The difference doesn't seem particularly significant -- both "what ifs" are equally impossible. "Regretting rationality" is on a different level -- rationality in the relevant sense is a matter of choice. The program that defines your decision-making algorithm isn't.

I still fear that you are reading in my words something very different from what I intend, as I don't see the possibility of a religious person's mind actually acting as i... (read more)

1Roko11ySo your argument is that the reason that the theists are wrong is because they only sorta-kinda believe in God anyway, but if they really believed, then they'd be just as right as we are?
2Vladimir_Nesov11yBut only in the sense that their calculation could be correct according to a particularly weird prior []. The difference between normal theist and a "god-impressed mind" who both believe in God is that of rationality: the former makes mistakes in updating beliefs, the latter probably doesn't. The same with an atheist god-impressed mind and a human atheist. You can't expect to find that weird a prior in a human. And of course, you should say that the god-impressed are wrong about their beliefs, though they correctly follow the evidence according to their prior. If you value their success in the real world more than the autonomy of their preference, you may want to reach into their minds and make appropriate changes. I should say again: the program that defines the decision-making algorithm can't be normally changed, which means that one can't be really "converted" to a different preference, though one can be converted to different beliefs and feelings. Observations don't change the algorithm, they are processed according to that algorithm. This means that if you care about reflective consistency (and everyone does, in the sense of preservation of preference), you'd try to counteract the unwanted effects of environment on yourself, including the self-promoting effects where you start liking the new situation. The extent to which you like the new situation, the "level of conviction", it's pretty much irrelevant, just as the presence of a losing psychological drive []. It'd take great integrity (not "strength of conviction") in the change for significantly different values to really sink in, in the sense that the new preference-on-reflection will resemble the new beliefs and feelings similarly to how the native preference-on-reflection will resemble native (sane, secular, etc.) beliefs and feelings.
1Roko11yI doubt that you can define a way to choose an algorithm out of a human brain that makes that sentence true.

I was just trying to clarify my interpretation of what you're saying. Because if they are theorizing of Xhi, if there are facts about Xhi and if they are seeking knowledge of it it seems clear that they ought to be doing science (in the general epistemological sense I was using earlier) to form these theories and discover these facts. This of course does not demonstrate that the two magesteria, as you've formulated them, are incoherent.

But I'm not sure if you are talking about the same thing Gould (and presumably Eliezer) are talking about. I took Gould t... (read more)

My intuition tells me that Level 4 is a mistake, and that there is such a thing as the consequence of my actions.

I disagree on the first part, and agree on the second part.

with the caveat that you can argue about weightings/priors over mathematical structures, so some consequences get a lower weighting than others, given the prior you chose.

Yes, and that's enough for rational decision making. I'm not really sure why you're not seeing that...

1Roko11yI agree that you can turn the handle on a particular piece of mathematics that resembles decisionmaking, but some part of me says that you're just playing a game with yourself: you decide that everything exists, then you put a prior over everything, then you act to maximize your utility, weighted by that prior. It is certainly a blow to one's intuition that one can only salvage the ability to act by playing a game of make-believe that some sections of "everything" are "less real" than others, where your real-ness prior is something you had to make up anyway. Others also think that I am just slow on the uptake of this idea. But to me the idea that reality is not fixed but relative to what real-ness prior you decide to pick is extremely ugly. It would mean that the utility of technology to achieve things is merely a shared delusion, that if a theist chose a real-ness prior that assigned high real-ness only to universes where a theistic god existed then he would be correct to pray, etc. Effectively you're saying that the postmodernists were right after all. Now, the fact that I have a negative emotional reaction to this proposal doesn't make it less true, of course.
1Vladimir_Nesov11yThere is a deep analogy between how you can't change the laws of physics (contents of reality, apart from lawfully acting) and how you can't change your own program. It's not a delusion unless it can be reached by mistake. The theist can't be right to act as if a deity exists unless his program (brain) is such that it is the correct way to act, and he can't change his mind for it to become right, because it's impossible to change one's program, only act according to it.

If you didn't imply

dualism requires us to give up rationality or science

and that

it follows that Bayesianism is wrong, that empiricism is wrong or that the scientific method is invalid in certain magesteria. (the coherence o one world view doesn't negate the others, and, anyway I haven't mentioned anything about a dichotomy between Bayesianism and dualism).

Then I am really confused by what you define as the dualism thesis in the original comment:

In this comment, by dualism, I'll mean the world view of two separate magisteria; one for science and o

... (read more)

You're correct of course.

But I'm 'sticking my neck out' on this one -- my intention was to signal this.

is often deliberately tricky

Then you're not really guessing at the truth of the statement. Instead you're guessing at the state of mind of the examiner, which involves a very different set of heuristics and biases.

Keep doing that and you may end up thinking, "maybe the correct answer is that he didn't defeat him because it was a Pyrrhic victory? Maybe Bosworth is spelled wrong? Maybe there's a dispute over the right year due to difficulties inherent in dating medieval events?"

Ah, I misunderstood the comment. I just assumed that Gallo was in on it, and the claim was that customers of Gallo failing to complain constituted evidence of wine tasting's crockitude.

If Gallo's wine experts really did get taken in, then yes, that's pretty strong evidence. And being the largest winery, I'm sure they have many experts checking their wines regularly--too many to realistically be "in" on such a scam.

So you've convinced me. Wine tasting is a crock.

What are some other seemingly "irrational" things we do that are in fact rational when we factor in the pleasantness of doing them?

Relevant paper: Lay Rationalism and Inconsistency between Predicted Experience and Decision

I know about the ultimatum game, but it is game-theoretically interesting precisely because the players have different preferences: I want all the money for me, you want all of it for you.

Ultimatum game was mentioned primarily to remind that the amount of FAI-value traded for assistance may be orders of magnitude greater than what the assistance feels to amount to.

We might as well have as a given that all the discussed values are (at least to some small extent) different. The "all of money" here are the points of disagreement, mutually exclusi... (read more)

2Roko11yThanks for the explanation. Another way to put this is that when people negotiate, they do best, all other things equal, if they try to drive a very hard bargain. If me and my neighbour Claire are both from roughly the same culture, upbringing, etc, and we are together going to build an AI which will extrapolate a combination of our volitions, Claire might do well to demand a 99% weighting to her volitions, and maybe I'll bargain her up to 90% or something. Bob the babyeater might offer me the same help that Claire could have given in exchange for just a 1% weighting of his volition, by the principle that I am making the same sacrifice in giving 99% of the CEV to Claire as in giving 1% to Bob. In reality, however, humans tend to live and work with people that are like them, rather than people who are unlike them. And the world we live in doesn't have a uniform distribution of power and knowledge across cultures. Many "alien" cultures are too powerless compared to ours to do anything. The However, China and India are potential exceptions. The USA and China may end up in a dictator game over FAI motivations. All I am saying is that the egalitarian desire to include all of humanity in CEV, each with equal weight, is not optimal. Yes dictator game/negotiation with China, yes dictator game/negotiation within US/EU/western block. Excluding a group from the CEV doesn't mean disenfranchising them. It means enfranchising them according to your definition of enfranchisement. Cultures in North Africa that genitally mutilate women should not be included in CEV, but I predict that my CEV would treat their culture with respect and dignity, including in some cases interfering to prevent them from using their share of the light-cone to commit extreme acts of torture or oppression.
2Vladimir_Nesov11yYou don't include cultures in CEV, you filter people through extrapolation of their volition. Even if culture makes value different, "mutilating women" is not a kind of thing that gets through, and so is a broken prototype example for drawing attention to. In any case, my argument in the above comment was that value should be given (theoretically, if everyone understands the deal and relevant game theory, etc., etc.; realistically, such a deal must be simplified; you may even get away with cheating) according to provided assistance, not according to compatibility of value. If poor compatibility of value prevents from giving assistance, this is an effect of value completely unrelated to post-FAI compatibility, and given that assistance can be given with money, the effect itself doesn't seem real either. You may well exclude people of Myanmar, because they are poor and can't affect your success, but not people of a generous/demanding genocidal cult, for an irrelevant reason that they are evil. Game theory is cynical.
2Roko11yhow do you know? If enough people want it strongly enough, it might.
1Vladimir_Nesov11yHow strongly people want something now doesn't matter, reflection has the power to wipe current consensus clean. You are not cooking a mixture of wants, you are letting them fight it out, and a losing want doesn't have to leave any residue. Only to the extent current wants might indicate extrapolated wants, should we take current wants into account.
2Roko11ySure. And tolerance, gender equality, multiculturalism, personal freedoms, etc might lose in such a battle. An extrapolation that is more nonlinear in its inputs cuts both ways.

Re: Your definitions.

You appear to be conflating ontological views (physicalism and dualism usually refer to these sorts of views, views about what kinds of things exist) with epistemological views. There is nothing in the definition of physicalism that requires us to have knowledge of the external world and nothing in dualism that requires us to give up rationality or science. You can be a physicalist and still think someone is deceiving your senses, for example. Also, this might just be me but 'materialism' should be jettisoned as outdated. "Materia... (read more)

I don't know who the authors were or the specific scientific subject matter of the paper. (I didn't need to know that to spot their misuse of statistics.)

Good point. Also, they might wish to rewrite and resubmit... in any case, you can't reveal anything they would want to lay original claim to of feel afraid of being scooped of.

Thanks for the pointer, but I think the argument you gave in that post is wrong. You argued that an agent smaller than the universe has to represent its goals using an approximate ontology (and therefore would have to later re-phrase its goals relative to more accurate ontologies). But such an agent can represent its goals/preferences in compressed form, instead of using an approximate ontology. With such compressed preferences, it may not have the computational resources to determine with certainty which course of action best satisfies its preferences, bu... (read more)

1Roko11yYes, if it has compressible preferences, which in reality is the case for e.g. humans and many plausible AIs. In reality problems of the form where you discover that your preferences are stated in terms of an incorrect ontology, e.g. souls, anticipated future experience, are where this really bites. I think that depends upon the structure of reality. Maybe there will be a series of philosophical shocks as severe as the physicality of mental states, Big Worlds, quantum MWI, etc. Suspicion should definitely be directed at what horrors will be unleashed upon a human or AI that discovers a correct theory of quantum gravity. Just as Big World cosmology can erode aggregative consequentialism, maybe the ultimate nature of quantum gravity will entirely erode any rational decision-making; perhaps some kind of ultimate ensemble theory already has. On the other hand, the idea of a one-time shock is also plausible.
3Wei_Dai11yThe reason I think it can just be a one-time shock is that we can extend our preferences to cover all possible mathematical structures. (I talked about this in Towards a New Decision Theory [].) Then, no matter what kind of universe we turn out to live in, whichever theory of quantum gravity turns out to be correct, the structure of the universe will correspond to some mathematical structure which we will have well-defined preferences over. I addressed this issue a bit in that post as well. Are you not convinced that rational decision-making is possible in Tegmark's Level IV Multiverse?
3Vladimir_Nesov11yThe next few posts on my blog are going to be basically about approaching this problem (and given the occasion, I may as well commit to writing the first post today). You should read [*] to get a better idea of why I see "preference over all mathematical structures" as a bad call. We can't say what "all mathematical structures" is, any given foundation only covers a portion of what we could invent. As the real world, mathematics that we might someday encounter can only be completely defined by the process of discovery (but if you capture this process, you may need nothing else). -- [*] S. Awodey (2004). `An Answer to Hellman's Question: 'Does Category Theory Provide a Framework for Mathematical Structuralism? []". Philosophia Mathematica 12(1):54-64.