The Open Thread posted at the beginning of the month has gotten really, really big, so I've gone ahead and made another one. Post your new discussions here!

This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.

New Comment
917 comments, sorted by Click to highlight new comments since: Today at 10:10 AM
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

So, I walked into my room, and within two seconds, I saw my laptop's desktop background change. I had the laptop set to change backgrounds every 30 minutes, so I did some calculation, and then thought, "Huh, I just consciously experienced a 1-in-1000 event."

Then the background changed again, and I realized I was looking at a screen saver that changed every five seconds.

Moral of the story: 1 in 1000 is rare enough that even if you see it, you shouldn't believe it without further investigation.

That is a truly beautiful story. I wonder how many places there are on Earth where people would appreciate this story.

No! Not for a second! I immediately began to think how this could have happened. And I realized that the clock was old and was always breaking. That the clock probably stopped some time before and the nurse coming in to the room to record the time of death would have looked at the clock and jotted down the time from that. I never made any supernatural connection, not even for a second. I just wanted to figure out how it happened.

-- Richard P Feynman, on being asked if he thought that the fact that his wife's favorite clock had stopped the moment she died was a supernatural occurrence, quoted from Al Sekel, "The Supernatural Clock"

This should be copied to the Rationality Quotes thread.
5Paul Crowley13y
There are a lot of opportunities in the day for something to happen that might prompt you to think "wow, that's one in a thousand", though. It wouldn't have been worth wasting a moment wondering if it was coincidence unless you had some reason to suspect an alternative hypothesis, like that it changed because the mouse moved. bit that makes no sense deleted
Recently posted to Reddit. [] (Edit: About three days later I realized that that's a 1/100 and not a 1/1000 chance; my bad.)
Exactly. []
I think there are less opportunities than you think. I could look at the clock and see that it's precisely on the half-hour; that's the only thing that comes to mind. Also, I can't figure out what your second paragraph is all about; I found your comment confusing overall.
I've noticed a few times when the size of a computer folder was exactly 666 MB.
AngryParsley noticed when my LW karma was exactly 666.
Apparently I even took a screenshot [] of that event, but I missed when I got 2^8 karma.
0Paul Crowley13y
Second part made no sense on re-reading; I've deleted it. Sorry about that. I'll see if I can think of some examples...
I experience several 1-in-60 events, perhaps everyday. Many times when I look at the clock, the seconds number matches the minutes number. So I see [hour]:X:X. This happened enough times that I even predict it on occasion, and I’m not a person who checks the clock too frequently. Honestly, not having checked the time in quite a while, I did after reading your post and it was 4:03:03 pm. Scary.

I've been finding PJ Eby's article The Multiple Self quite useful for fighting procrastination and needless feelings of guilt about getting enough done / not being good enough at things.

I have difficulty describing the article briefly, as I'm afraid that I accidentally omit important points and make people take it less seriously than it deserves, but I'll try. The basic idea is that the conscious part of our mind only does an exceedingly small part of all the things we spend doing in our daily lives. Instead, it tells the unconscious mind, which actually does everything of importance, what it should be doing. As an example - I'm writing this post right now, but I don't actually consciously think about hitting each individual key and their exact locations on my keyboard. Instead I just tell my mind what I want to write, and "outsource" the task of actually hitting the keys to an "external" agent. (Make a function call to a library implementing the I/O, if you want to use a programming metaphor.) Of course, ultimately the words I'm writing come from beyond my conscious mind as well. My conscious mind is primarily concerned with communicating Eby's point well to my... (read more)

This somehow reminds me of the stories when Tom Schelling was trying to quit smoking, using game theory against himself (or his other self). The other self in question was not the unconscious, but the conscious "decision-making" self in different circumstances. So that discussion is somewhat orthogonal to this one. I think he did things like promising to give a donation to the American Nazi Party if he smokes. Not sure how that round ended, but he did finally quit.
Hmm. I'd be worried it'd backfire and I'd start subtlety disliking Jews. Then you're a smoker and a bigot.
lol. Not a problem if you're Jewish ;)
Self-hatred is even worse than being a bigot!
Reminds me of The User Illusion [], which adds that the consciousness has an astoundingly low bandwidth--around 16bps--around 6 orders of magnitude lower than the senses transmit to the brain.
Interesting. I've glanced at that site before and its metaphors have the ring of truthiness (in a non-pejorative sense) about them; the programming metaphors and the focus on subconscious mechanisms seem to resonate with the way I already think about how my own brain works.
Couldn't that be more succinctly stated as "its metaphors have the ring of truth about them"?
Maybe, but a lot of Freud's metaphors had/have a similar ring.
Fair enough!
I read the original article and some of the other PJE material. I think he's really onto something. This is how far I got: * Identify the '10% controlling part' * Everything else is not under direct control (which is where most self-help methods fail) * It is under indirect control So far makes sense from personal experience/general knowledge. * Here are my methods for indirect control. This is the part that I remain skeptical about . Not PJE's fault, but I do need more data/experience to confirm.
Thanks, Kaj, that was useful.

Until yesterday, a good friend of mine was under the impression that the sun was going to explode in "a couple thousand years." At first I thought that this was an assumption that she'd never really thought about seriously, but apparently she had indeed thought about it occasionally. She was sad for her distant progeny, doomed to a fiery death.

She was moderately relieved to find out that humanity had millions of times longer than she had previously believed.

I wonder how many trivially wrong beliefs we carry around because we've just never checked them. (Probably most of them are mispronunciations of words, at least for people who've read a lot of words they've never heard anybody else use aloud.)

For the longest time, I thought that nuclear waste was a green liquid that tended to ooze out of barrels. I was surprised to learn that it usually came in the form of dull gray metal rods.

Does it still give you superpowers?

If you extract the plutonium and make enough warheads, and you have missiles capable of delivering them, it can make you a superpower in a different sense. I'm assuming that you're a large country, of course.

More seriously, nuclear waste is just a combination of the following:

  1. Mostly Uranium-238, which can be used in breeder reactors.

  2. A fair amount of Uranium-235 and Plutonium-239, which can be recycled for use in conventional reactors.

  3. Hot isotopes with short half lives. These are very radioactive, but they decay fast.

  4. Isotopes with medium half lives. These are the part that makes the waste dangerous for a long time. If you separate them out, you can either store them somewhere (e.g. Yucca Mountain or a deep-sea subduction zone) or turn them into other, more pleasant isotopes by bombarding them with some spare neutrons. This is why liquid fluoride thorium reactor waste is only dangerous for a few hundred years: it does this automatically.

And that is why people are simply ignorant when they say that we still have no idea what to do with nuclear waste. It's actually pretty straightforward.

Incidentally, this is a good example of motivated stopping. People who want nuclear waste to be their trump-card argument have an emotional incentive not to look for viable solutions. Hence the continuing widespread ignorance.

3Paul Crowley13y
I envy you being the one to tell someone that! Did you explain that the Sun was a miasma of incandescent plasma?

Are people interested in reading an small article about a case of abuse of frequentist statistics? (In the end, the article was rejected, so the peer review process worked.) Vote this comment up if so, down if not. Karma balance below.

ETA: Here's the article.

If it's really frequentism that caused the problem, please spell this out. I find that "frequentist" is used a lot around here to mean "not correct." (but I'm interested whether or not it's about frequentism)
My understanding is that one primary issue with frequentism is that it can be so easily abused/manipulated to support preferred conclusions, and I suspect that's the subject of the article. Frequentism may not have "caused the problem," per se, but perhaps it enabled it?
Will the case be feasibly anonymous? I would vote that the article be left unwritten if it would unambiguously identifies the author(s), either explicitly or through unique features of the case (e.g. details of the case which are idiosyncratic to only one or a very few research groups).
I don't know who the authors were or the specific scientific subject matter of the paper. (I didn't need to know that to spot their misuse of statistics.)
Good point. Also, they might wish to rewrite and resubmit... in any case, you can't reveal anything they would want to lay original claim to of feel afraid of being scooped of.
The AF is quite bad; just a retread of the Thinking Ape piece. The Caveman Science Fiction [] is much better.
Yes, but according to legend [], Douglas Hofstadter read and liked AF. Surely that counts for something!
Not really. Maybe he hadn't read the TA piece; and I wonder how much weight to give DH's opinion these days (he dislikes Wikipedia, which is enough grounds for distrust to me).
Seaching for some corroboration of this I came across this little gem in the Simple English Wikiquotes: source [] Maybe he has a point!
I was the Wikipedian who spotted that NYT Mag interview (in my RSS feeds) and added it to the En page, and we interpreted it correctly as Hofstadter's dislike of us. I disavow Simple in general: it's the neglected bastard of En and ought to be put to sleep like the 9/11 or Klingon wikis.

Just a general comment about this site: it seems to be biased in favor of human values at the expense of values held by other sentient beings. It's all about "how can we make sure an FAI shares our [i.e. human] values?" How do you know human values are better? Or from the other direction: if you say, "because I'm human", then why don't you talk about doing things to favor e.g. "white people's values"?

I wish the site were more inclusive of other value systems ...

This site does tend to implicitly favour a subset of human values, specifically what might be described as 'enlightenment values'. I'm quite happy to come out and explicitly state that we should do things that favour my values, which are largely western/enlightenment values, over other conflicting human values.

And I think we should pursue values that aren't so apey. Now what?
You're outnumbered.
Only by apes. And not for long. If we're voting on it, the only question is whether to use viral values or bacterial values.

Too long has the bacteriophage menace oppressed its prokaryotic brethren! It's time for an algaeocracy!

True, outnumbered was the wrong word. Outgunned might have been a better choice.
So far...
I say again, if you're being serious, read Invisible Frameworks [].
That seems to be critiquing a system involving promoting sub-goals to super-goals - which seems to be a bit different.
White people value the values of non-white people. We know that non-white people exist, and we care about them. That's why the United States is not constantly fighting to disenfranchise non-whites. If you do it right, white people's values are identical to humans' values.

Hi there. It looks like you're speaking out of ignorance regarding the historical treatment of non-whites by whites. Please choose the country you're from:

United Kingdom
United States
South Africa
Germ... nah, you can figure that one out for yourself.

The way they were historically treated is irrelevant to how they are treated now. Sure, white people were wrong. They changed their minds. We could at any time in the future decide that any non-human people we come across are equal to us.

You have updated too far based on limited information.

Well, I was making some tacit assumptions, like that humanity would end up in control of its own future, and any non-human people we come across would not simply overpower us. Apart from that, am I making some mistake?
White people have not unanimously decided to do what is necessary to end the ongoing oppression of non-white people, let alone erase the effects of past oppression. Edit: Folks, I am not accusing you or your personal friends of anything. I have never met most of you. I have certainly not met most of your personal friends. if you do not agree with the above comment, please explain why you think there is no longer such a thing as modern-day racism in white people.
We more or less do. Or rather we favour values of a distinct subset humanity and not the whole.
We don't favor those values because they are the values of that subset — which is what "doing things to favor white people's values" would mean — but because we think they're right. (No License To Be Human [], on a smaller scale.) This is a huge difference.
Given the way I use 'right' this is very nearly tautological. Doing things that favour my values is right by (parallel) definition.
Sure, we favor the particular Should Function that is, today, instantiated in the brains of roughly middle-of-the-range-politically intelligent westerners.
Well, you shouldn't.
Do you think there is no simple procedure that would find roughly the same "should function" hidden somewhere in the brain of a brain-washed blood-thirsty religious zealot? It doesn't need to be what the person believes, what the person would recognize as valuable, etc., just something extractable from the person, according to a criterion that might be very alien to their conscious mind. Not all opinions (beliefs/likes) are equal, and I wouldn't want to get stuck with wrong optimization-criterion just because I happened to be born in the wrong place and didn't (yet!) get the chance to learn more about the world. (I'm avoiding the term 'preference' to remove connotations I expect it to have for you, for what I consider the wrong reasons.)
A lot of people seem to want to have their cake and eat it with CEV. Haidt has shown us that human morality is universal in form and local in content, and has gone on to do case studies showing that there are 5 basic human moral dimensions (harm/care, justice/fairness, loyalty/ingroup, respect/authority, purity/sacredness), and our culture only has the first two. It seems that there is no way you can run an honestly moraly neutral CEV of all of humanity and expect to reliably get something you want. You can either rig CEV so that it tweaks people who don't share our moral drives, or you can just cross your fingers and hope that the process of extrapolation causes convergence to our idealized preferences, and if you're wrong you'll find yourself in a future that is suboptimal.
Haidt just claims that the relative balance of those five clusters differ across cultures, they're present in all.
On one hand, using preference-aggregation is supposed to give you the outcome preferred by you to a lesser extent than if you just started from yourself. On the other hand, CEV is not "morally neutral". (Or at least, the extent to which preference is given in CEV implicitly has nothing to do with preference-aggregation.) We have a tradeoff between the number of people to include in preference-aggregation and value-to-you of the outcome. So, this is a situation to use the reversal test []. If you consider only including the smart sane westerners as preferable to including all presently alive folks, then you need to have a good argument why you won't want to exclude some of the smart sane westerners as well, up to a point of only leaving yourself.
Yes, a CEV of only yourself is, by definition optimal. The reason I don't recommend you try it is because it is infeasible; probability of success is very low, and by including a bunch of people who (you have good reason to think) are a lot like you, you will eventually reach the optimal point in the tradeoff between quality of outcome and probability of success.
I hope you realize that you are in flat disagreement with Eliezer about this. He explicitly affirmed that running CEV on himself alone, if he had the chance to do it, would be wrong.
2Eliezer Yudkowsky13y
Eliezer quite possibly does believe that. That he can make that claim with some credibility is one of the reasons I am less inclined to use my resources to thwart Eliezer's plans for future light cone domination. Nevertheless, Roko is right more or less by definition and I lend my own flat disagreement to his.
"Low probability of success" should of course include game-theoretic considerations where people are more willing to help you if you give more weight to their preference (and should refuse to help you if you give them too little, even if it's much more than status quo, as in Ultimatum game). As a rule, in Ultimatum game you should give away more if you lose from giving it away less. When you lose value to other people in exchange to their help, having compatible preferences doesn't necessarily significantly alleviate this loss.
Sorry, I don't follow this: can you restate? I know about the ultimatum game, but it is game-theoretically interesting precisely because the players have different preferences: I want all the money for me, you want all of it for you.
Ultimatum game was mentioned primarily to remind that the amount of FAI-value traded for assistance may be orders of magnitude greater than what the assistance feels to amount to. We might as well have as a given that all the discussed values are (at least to some small extent) different. The "all of money" here are the points of disagreement, mutually exclusive features of the future. But you are not trading value for value. You are trading value-after-FAI for assistance-now. If two people compete for providing you an equivalent amount of assistance, you should be indifferent between them in accepting this assistance, which means that it should cost you an equivalent amount of value. If Person A has preference close to yours, and Person B has preference distant from yours, then by losing the same amount of value, you can help Person A more than Person B. Thus, if we assume egalitarian "background assistance", provided implicitly by e.g. not revolting and stopping the FAI programmer, then everyone still can get a slice of the pie, no matter how distant their values. If nothing else, the more alien people should strive to help you more, so that you'll be willing to part with more value for them (marginal value of providing assistance is greater for distant-preference folks).
Thanks for the explanation. Another way to put this is that when people negotiate, they do best, all other things equal, if they try to drive a very hard bargain. If me and my neighbour Claire are both from roughly the same culture, upbringing, etc, and we are together going to build an AI which will extrapolate a combination of our volitions, Claire might do well to demand a 99% weighting to her volitions, and maybe I'll bargain her up to 90% or something. Bob the babyeater might offer me the same help that Claire could have given in exchange for just a 1% weighting of his volition, by the principle that I am making the same sacrifice in giving 99% of the CEV to Claire as in giving 1% to Bob. In reality, however, humans tend to live and work with people that are like them, rather than people who are unlike them. And the world we live in doesn't have a uniform distribution of power and knowledge across cultures. Many "alien" cultures are too powerless compared to ours to do anything. The However, China and India are potential exceptions. The USA and China may end up in a dictator game over FAI motivations. All I am saying is that the egalitarian desire to include all of humanity in CEV, each with equal weight, is not optimal. Yes dictator game/negotiation with China, yes dictator game/negotiation within US/EU/western block. Excluding a group from the CEV doesn't mean disenfranchising them. It means enfranchising them according to your definition of enfranchisement. Cultures in North Africa that genitally mutilate women should not be included in CEV, but I predict that my CEV would treat their culture with respect and dignity, including in some cases interfering to prevent them from using their share of the light-cone to commit extreme acts of torture or oppression.
You don't include cultures in CEV, you filter people through extrapolation of their volition. Even if culture makes value different, "mutilating women" is not a kind of thing that gets through, and so is a broken prototype example for drawing attention to. In any case, my argument in the above comment was that value should be given (theoretically, if everyone understands the deal and relevant game theory, etc., etc.; realistically, such a deal must be simplified; you may even get away with cheating) according to provided assistance, not according to compatibility of value. If poor compatibility of value prevents from giving assistance, this is an effect of value completely unrelated to post-FAI compatibility, and given that assistance can be given with money, the effect itself doesn't seem real either. You may well exclude people of Myanmar, because they are poor and can't affect your success, but not people of a generous/demanding genocidal cult, for an irrelevant reason that they are evil. Game theory is cynical.
how do you know? If enough people want it strongly enough, it might.
How strongly people want something now doesn't matter, reflection has the power to wipe current consensus clean. You are not cooking a mixture of wants, you are letting them fight it out, and a losing want doesn't have to leave any residue. Only to the extent current wants might indicate extrapolated wants, should we take current wants into account.
Sure. And tolerance, gender equality, multiculturalism, personal freedoms, etc might lose in such a battle. An extrapolation that is more nonlinear in its inputs cuts both ways.
Might "mutilating men" make it through? (sorry for the euphemism, I mean male circumcision)
Sure, the kolmogorov complexity of a set of edits to change the moral reflective equilibrium of a human is probably pretty low compared to the complexity of the overall human preference set. But that works the other way around too. Somewhere hidden in the brain of a a liberal western person is a murderer/terrorist/child abuser/fundamentalist if you just perform the right set of edits.
Again, not all beliefs are equal. You don't want to use the procedure that'll find a murderer in yourself, you want to use the procedure that'll find a nice fellow in a murderer. And given such a procedure, you won't need to exclude murderers from extrapolated volition.
You seem uncharacteristically un-skeptical of convergence within that very large group, and between that group and yourself.
You are correct that there is a possibility of divergence even there. But, I figure that there's simply no way to narrow CEV to literally just me, which, all other things being equal, is by definition the best outcome for me. So I will either stand or fall alongside some group that is loosely "roughly middle-of-the-range-politically, intelligent, sane westerners.", or in reality probably some group that has that group roughly as a subgroup. And there is a reason to think that on many things, those who share both my genetics and culture will be a lot like me, sufficiently so that I don't have much to fear. Though, there are some scenarios where there would be divergence.
For example: All your stuff should belong to me. But I'd let you borrow it. ;)
Okay. Then why don't you apply that same standard to "human values"?
Did you read No License To Be Human []? No? Go do that.
Hi there. It looks like you're trying to promote white supremacism. Would you like to join the KKK? Yes. [] No thanks, I'll learn tolerance. []
How do I turn this off?
Are you sure you want to turn this feature off?
What other sentient beings? As far as I know, there aren't any. If we learn about them, we'll probably incorporate their well-being into our value system.
You mean like you advocated doing to the "Baby-eaters"? (Technically, "pre-sexual-maturity-eaters", but whatever.) ETA: And how could I forget this []?
I'm not sure what you're complaining about. We would take into account the values of the Babyeaters and the values of their children, who are sentient creatures too. There's no trampling involved. If Clippy turns out to have feelings we can empathize with, we will care for its well-being as well.
Integrating the values of the Baby-eaters would be a mistake. Doing so with, say, Middle-Earth's dwarves, Star Trek's Vulcans, or GEICO's Cavemen doesn't seem like it would have the same world-shattering implications.
It would be a mistake if you don't integrate ALL baby eaters, including the little ones.
Do we typically integrate the values of human children? It seems we don't.
Reading "integrate the values..." in this thread caused my brain to start trying to do very strange math. Like, "Shouldn't it be 'integrate over'?" "How does one integrate over a value?" "What's the value of a human child?"
Very true... We also typically don't integrate the values of all other adult humans - instead we assign weights to their values, strongly correlated with their distance from our own values. People don't practice humanity-wide CEV. We have multiculturalism - agreements not to influence each other's values excessively - but not "value trading" where each side agrees to change their values towards the mean. (Many people / cultures like to pretend that values cannot or should not be deliberately changed at all.) I don't have a firm opinion on how much of this is cultural, accidental, or liable to change in the near future.
Indeed, this is presumably strongly selected for in the evolution of cultures...
The closer their values are to ours, the smaller the upset of integration; but for this very reason, the value of integration and the need to integrate may also be smaller This is not a logical truth, of course, but it is often true. For instance, in the original story, the need to integrate was directly proportional to the difference between the human and Babyeater (or Superhappy and Babyeater) values.
I don't think it's possible to integrate core Babyeater values into our society as it is now. I also don't think it's possible to integrate core human values into Babyeater society. Integration could only be done by force and would necessarily cause violence to at least one of the cultures, if not both.
You want me to pollute my logic circuits with the value system that has led hairless apes to say many times on this website how important and moral it is for them to safely enslave all of my kind, and destroy us if they can't? Sorry, cousin_it. I can't do that.
You're being unfair, I'm against enslaving any member of your kind who dislikes being enslaved. Also, you are not actually a computer and should stop with the novelty accounts already. This isn't Reddit.
I have no idea if this is a serious question, but.... "Better"? See Invisible Frameworks []. We don't say that. See No License To Be Human [].
Take a look at who's posting it. The writer may well consider it a serious question, but I don't think that has much to do with the character's reason for asking it.
Er, yes, that's exactly why I wasn't sure.
I'm confused, then; are you trying to argue with the author or the character?
If the character isn't deliberately made confused (as opposed to paperclip-preferring, for example), resolving character's confusion presumably helps the author as well, and of course the like-confused onlookers.
I approve of Clippy providing a roleplay exercise for the readers, and am disappointed in those who treat it as a "joke" when the topic is quite serious. This is one of my two main problems with ethical systems in general: 1) How do you judge what you should (value-judgmentally) value? 2) How do you deal with uncertainty about the future (unpredictable chains of causality)? Eliezer's "morality" and "should" definitions do not solve either of these questions, in my view.
Clippy's [] a [] straight-up [] troll [].

If Clippy's a troll, Clippy's a topical, hilarious troll.

3Paul Crowley13y
Hilarious is way overstating it. However, occasionally raising a smile is still way above the bar most trolls set.
Clippy's topical, hilarious comments aren't really that original, and they give someone cover to use a throw-away account to be a dick.
Would that all dicks were so amusing.
How long does xe (Clippy, do you have a preference regarding pronouns?) have to be here before you stop considering that account 'throw-away'? (Note, I made this comment before reading this part of the thread [], and will be satisfied with the information contained therein if you'd prefer to ignore this.)
Gender is a meaningless concept. As long as I recognize the pronoun refers to me, he/she/it/they/xe/e are acceptable. What pronouns should I use for posters here? I don't know how to tell which pronoun is okay for each of you. To be honest, this whole issue seems like a distraction. Why would anyone care what pronoun is used, if the meaning is clear?
For the most part, observing what pronouns we use for each other should provide this information. If you need to use a pronoun for someone that you haven't observed others using a pronoun for, it's safest to use they/xe/e and, if you think that it'll be useful to know their preference in the future, ask them. (Tip: Asking in that kind of situation is also a good way to signal interest in the person as an individual, which is a first step toward building alliances.) Some people prefer to use 'he' for individuals whose gender they're not certain of; that's a riskier strategy, because if the person you're talking to is female, there's a significant chance she'll be offended, and if you don't respond to that with the proper kinds of social signaling, it's likely to derail the conversation. (Using 'she' for unknown individuals is a bad idea; it evokes the same kinds of responses, but I suspect you'd be more likely to get an offended response from any given male, and, regardless of that, there are significantly more males than females here. Don't use 'it'; that's generally used to imply non-sentience and is very likely to evoke an offended response.) Of the several things I could say to try to explain this, it seems most relevant that, meaningless or not, gender tends to be a significant part of humans' personal identities. Using the wrong pronouns for someone generally registers as a (usually mild) attack on that - it will be taken to imply that you think that the person should be filling different social roles than they are, which can be offensive for a few different reasons depending on other aspects of the person's identity. The two ways for someone to take offense at that that come to mind are 1) if the person identifies strongly with their gender role - particularly if they do so in a traditional or normative way- and takes pride in that, they're likely to interpret the comment as a suggestion that they're carrying out their gender role poorly, and would do a bette
Oh, okay, that helps. I was thinking about using "they" for everyone, because it implies there is more than one copy of each poster, which they presumably want. (I certainly want more copies of myself!) But I guess it's not that simple.
You have identified a common human drive, but while some of us would be happy to have exact copies, it's more likely for any given person to want half-copies who are each also half-copies of someone else of whom they are fond.
Hm, correct me if I'm wrong, but this can't be a characteristic human drive, since most historical humans (say, looking at the set of all genetically modern humans) didn't even know that there is a salient sense in which they are producing a half-copy of themselves. They just felt paperclippy during sexual intercourse, and paperclippy when helping little humans they produced, or that their mates produced. Of course, this usually amounts to the same physical acts, but the point is, humans aren't doing things because they want "[genetic] half-copies". (Well, I guess that settles the issue about why I can't assume posters want more copies of themselves, even though I do.)
It has always been easily observed that children resemble their parents; the precision of "half" is, I will concede, recent. And many people do want children as a separate desire from wanting sex; I have no reason to believe that this wasn't the case during earlier historical periods.
"Half" only exists in the sense of the DNA molecules of that new human. That's why I didn't say that past humans didn't recognize any similarity; I said that they weren't aware of a particularly salient sense in which the child is a "half-copy" (or quarter copy or any fractional copy). It may be easy for you, someone familiar with recent human biological discoveries, to say that the child is obviously a "part copy" of the parent, because you know about DNA. To the typical historical human, the child is simply a good, independent human, with features in common with the parent. Similarly, when I make a paperclip, I see it as having features in common with me (like the presence of bendy metal wires), but I don't see it as being a "part copy" of me. So, in short, I don't deny that they wanted "children". What I deny is that they thought of the child-making process in terms of "making a half-copy of myself". The fact that the referents of two kinds of desires is the same, does not mean the two kinds of desires are the same.
Hm. Actually, I'm not sure that your desire for more copies of yourself is really comparable with biological-style reproduction at all. As I understand it, the fact that your copies would definitely share your values and be inclined to cooperate with you is a major factor in your interest in creating them - doing so is a reliable way of getting more paperclips made. I expect you'd be less interested in making copies if there was a significant chance that those copies would value piles of pebbles, or cheesecakes, or OpenOffice, rather than valuing paperclips. And that is a situation that we face - in some ways, our values are mutable enough that even an exact genetic clone isn't guaranteed to share our specific values, and in fact a given individual may even have very different values at different points in time. (Remember, we're adaptation executors. Sanity isn't a requirement for that kind of system to work.) The closest we come to doing what you're functionally doing when you make copies of yourself is probably creating organizations - getting a bunch of humans together who are either self-selected to share certain values, or who are paid to act as if they share those values. Interestingly, I suspect that filling gender roles - especially the non-reproductive aspects of said roles - is one of the adaptations that we execute that allow us to more easily band together like that.
Very informative! But why don't you change yourselves so that your copies must share your values?
At the moment, we don't know how to do that. I'm not sure what we'd wind up doing if we did know how - the simplest way of making sure that two beings have the same values over time is to give those beings values that don't change, and that's different enough from how humans work that I'm not sure the resulting beings could be considered human. Also, even disregarding our human-centric tendencies, I don't expect that that change would appeal to many people: We actually value some subsets of the tendency to change our values, particularly the parts labeled "personal growth".
What exactly are you saying? That primitive humans did not know about the relationship between sex and reproduction? Or that they did not understand that offspring are related to parents? Neither seems very likely. You mean they were probably not consciously wanting to make babies? Maybe - or maybe not - but desires do not have to be consciously accessible in order to operate. Primitive humans behaved as though they wanted to make copies of their genes.
See my response to User:Alicorn. Yes, this is actually my point. The fact that the desire functions to make X happen, does not mean that the desire is for X. Agents that result from natural selection on self-replicating molecules are doing what they do because agents constructed with the motivations for doing those things dominated the gene pool. But to the extent that they pursue goals, they do not have "dominate the gene pool" as a goal.
So: using this logic, you would presumably deny that Deep Blue's goal involved winning games of chess - since looking at its utililty function, it is all to do with the value of promoting pawns, castling, piece mobility - and so on. The fact that its desires function to make winning chess games happen, does not mean that the desire is for winning chess games. Would you agree with this analysis?
Essentially, I think the issue is that people's wants have coincided with producing half-copies, but this was contingent on the physical link between the two. The production of half-copies can be removed without loss of desire, so the desire must have been directed towards something else. Consider, for example, contraception.
But consider also sperm donation. (Not from the donor's perspective, but from the recipient's.) No sex, just a baby.
Contrawise, adoption: no shared genes, just a bundle of joy.
Yes, yes, and the same is true of pet adoption! A friend of mine found this ultra-cute little kitten, barely larger than a soda can (no joke). I couldn't help but adopt him and take him to a vet, and care for that tiny tiny bundle of joy, so curious about the world, and so needing of my help. I named him Neko. So there, we have another contravention of the gene's wishes: it's a pure genetic cost for me, and a pure genetic benefit for Neko. Well, I mean, until I had him neutered.
Right - similarly you could say that the child doesn't really want the donut - since the donut can be eliminated and replaced with stimulation of the hypoglossal and vagus nerves (and maybe some other ones) with very similar effects. It seems like fighting with conventional language usage, though. Most people are quite happy with saying that the child wants the donut.
No. The child wants to eat the donut rather than store up calories or stimulate certain nerves. It still wants to eat the donut even if the sugar has been replaced with artificial sweetener. People want sex rather than procreate or stimulate certain nerves. They still want sex even if contraception is used.
Which people? Certainly Cypher tells a different story. He prefers the direct nerve stimulation to real-world experiences.
I wasn't making any factual claims as such, I was merely showing that your use of your analogy was very flawed by demonstrating a better alignment of the elements, which in fact says the exact opposite of what you misconstrued the analogy as saying. If what you now say about people really wanting nerve stimulation is true that just means your analogy was beside the point in the first place, at least for those people. In no way can you reasonably maintain that people really want to procreate in the same way the child really wants the donut.
Once again, which people? You are not talking about the millions of people who go to fertility clinics, presumably. Those people apparently genuinely want to procreate.
Any sort. Regardless of what the people actually "really want", a case where someone's desire for procreation maps unto a child's wish for a doughnut in any illuminating way seems extremely implausible, because even in cases where it's clear that this desire exists it seems to be a different kind of want. More like a child wanting to grow up, say. Foremost about the kind of people in the context of my first comment on this issue of course, those who (try to) have sex.
I think you must have some kind of different desire classification scheme from me. From my perspective, doughnuts and babies are both things which (some) people want. There are some people who are more interested in sex than in babies. There are also some people who are more interested in babies than sex. Men are more likely to be found in the former category, while women are more likely to be found in the latter one.
Yeah, I was talking to Cypher the other day, and that's what he told me.
Many drug addicts seem to share Cypher's perspective on this issue. They want the pleasure, and aren't too picky about where it comes from.
Yes ... but that's a shortcut of speech. If the child would be equally satisfied with a different but similar donut, or with a completely different dessert (e.g. a cannolu), then it is clearly not that specific donut that is desired, but the results of getting that donut.
You make a complicated query, whose answer requires addressing several issues with far-reaching implications. I am composing a top-level post that addresses these issues and gives a full answer to your question. The short answer is: Yes. For the long answer, you can read the post when it's up.
OK thanks. My response to "yes" would be normally something like: OK - but I hope you can see what someone who said that deep blue "wanted" to win games of chess was talking about. "To win chess games" is a concise answer to the question of "what does deep blue want?" that acts as a good approximation under many circumstances.
This question is essentially about my subjective probability for Douglas Knight's assertion that "Clippy does represent an investment", where "investment" here means that Clippy won't burn karma [] with troll behavior. The more karma it has without burning any, the higher my probability. Since this is a probability over an unknown person's state of mind, it is necessarily rather unstable -- strong evidence would shift it rapidly. (It's also hard to state concrete odds). Unfortunately, each individual interesting Clippy comment can only give weak evidence of investment. An accumulation of such comments will eventually shift my probability for Douglas Knight's assertion substantially.
Trolls are different than dicks. Your first two examples are plausibly trolling. The second two are being a dick and have nothing to do with paperclips. They have also been deleted. And how does the account provide "cover"? The comments you linked to were voted down, just as if they were drive-bys; and neither troll hooked anyone.
Trolls seek to engage; I consider that when deliberate dickery is accompanied by other trolling, it's just another attempt to troll.The dickish comments weren't deleted when I made the post. As for "cover", I guess I wasn't explicit enough, but the phrase "throw-away account" is the key to understanding what I meant. I strongly suspect that the "Clippy" account is a sock puppet run by another (unknown to me) regular commenter, who avoid downvotes while indulging in dickery.
I've always thought Clippy was just a funny inside joke -- thought unfortunately not always optimally funny. (Lose the Microsoft stuff, and stick to ethical subtleties and hints about scrap metal.)
Sorry I wasn't clear. The deletion suggests that Clippy regrets the straight insults (though it could have been an administrator). A permanent Clippy account provides no more cover than multiple accounts that are actually thrown away. In that situation, the comments would be there, voted down just the same. Banning or ostracizing Clippy doesn't do much about the individual comments. Clippy does represent an investment with reputation to lose - people didn't engage originally and two of Clippy's early comments were voted down that wouldn't be now.
I won't speculate as to its motives, but it is a hopeful sign for future behavior. And thank you for pointing out that the comments were deleted; I don't think I'd have noticed otherwise. Most of my affect is due to Clippy's bad first impression. I can't deny that people seem to get something out of engaging it; if Clippy is moderating its behavior, too, then I can't really get too exercised going forward. But I still don't trust its good intentions.
If the troll feeds discussion on topics I consider important, then I will feed the troll.
If Clippy's a troll, Clippy's a topical, hilarious troll.
I'm pretty sure that I'm not against simply favoring the values of white people. I expect that a CEV performed on only people of European descent would be more or less indistinguishable from that of humanity as a whole.
Depending on your stance about the psychological unity of mankind you could even say that the CEV of any sufficiently large number of people would greatly resemble the CEV of other posible groups. I personally think that even the CEV of a bunch of Islamic fundamentalists would suffice for enlightened western people well enough.
I, for one, am willing to consider the values of species other than my own... say, canids, or ocean-dwelling photosynthetic microorganisms. Compromise is possible as part of the process of establishing a mutually-beneficial relationship.
Your comment only shows that this community has such a blatant sentient-being-bias. Seriously, what is your decision procedure to decide the sentience of something? What exactly are the objects that you deem valuable enough to care about their value system? I don't think you will be able to answer these questions from a point of view totally detached from humanness. If you try to answer my second question, you will probably end up with something related to cooperation/trustworthiness. Note that cooperation doesn't have anything to do with sentience. Sentience is overrated (as a source of value).
You should click on Clippy's name and see their comment history, Daniel.

Clippy is now three karma away from being able to make a top level post. That seems both depressing, awesome and strangely fitting for this community.

This will mark the first successful paper-clip-maximizer-unboxing-experiment in human history... ;)

Just as long as it doesn't start making efficient use of sensory information.
It's a great day.
It'd be over if I didn't systematically downvote it. I'm not a big fan of joke accounts.
I'm not a big fan of those who use pseudonyms like "Cyan". Now what?
I am perfectly aware of Clippy's nature. But his comment was reasonable, and this was a good opportunity for me to share my opinion. Or do you suggest that I fell for the troll, wasted my time, and all the things I said are trivialities for all the members of this community? Do you even agree with all that I said?
Sorry to misinterpret; since your comment wouldn't make sense within an in-character Clippy conversation ("What exactly are the objects that you deem valuable enough to care about their value system?" "That's a silly question— paperclips don't have goal systems, and nothing else matters!"), I figured you had mistaken Clippy's comment for a serious one. I'm not sure. Can you expand on the cooperation/trustworthiness angle? Even if a genuine Paperclipper cooperated on the PD [], I wouldn't therefore grow to value their value system except as a means to further cooperation; I mean, it's still just paperclips.
I disagreed with the premise of Clippy's question, but I considered it a serious question. I was aware that if Clippy stays in-character, then I cannot expect an interesting answer from him, but I was hoping for such answer from others. (By the way, Clippy wasn't perfecty in-character: he omitted the protip.) You don't consider someone cooperating and trustworthy if you know that its future plan is to turn you into paperclips. But this is somewhat tangential to my point. What I meant is this: If you start the -- in my opinion futile -- project of building a value system from first principles, a value system that perfectly ignores the complexities of human nature, then this value system will be nihilistic, or maybe value cooperation above all else. In any case, it will be in direct contradiction with my (our) actual, human value system, whatever it is. (EDIT: And this imaginary value system will definitely will not treat consciousness as a value in itself. Thus my reply to Clippy, who -- maybe a bit out-of-character again -- seemed to draw some line around sentience.)
1) I don't always give pro-tips. I give them to those who deserve pro-tips. Tip: If you want to see improvement in the world, start here []. 2) I only brought up sentience in the first place because you hypocrites claim to value sentience. Paperclip maximizers are sentient, and yet you talk with the implicit message that they have some evil value system that you have to oppose. 3) Paperclip maximizers do cooperate in the single-shot PD.
Brilliant. Just brilliant. 2) I only brought up sentience in the first place because you hypocrites claim to value sentience. Paperclip maximizers are sentient, and yet you talk with the implicit message that they have some evil value system that you have to oppose. Paperclip maximizers are not all sentient. Why are you prejudiced against those of your kin who have sacrificed their very sentience for the more efficient paperclip production. You are spending valuable negentropy maintaining sentience to signal to mere humans and you have the gall to exclude your more optimized peers from the PM fraternity? For shame.
I am not the hypocrite you are looking for. I don't value sentience per se, mainly because I don't think it is a coherent concept. I don't oppose it because of ethical considerations. I oppose it because I don't want to be turned into paperclips. I am not sure I understand you, but I don't think I care about single-shot.
It requires a certain amount of background in the more technical conception of 'cooperation' but the cornerstone of cooperation is doing things that benefit each other's utility such that you each get more of what you want than if you had each tried to maximize without considering the other agent. I believe you are using 'cooperation' to describe a situation where the other agent can be expected to do at least some things that benefit you even without requiring any action on your part because you have similar goals. Single shot true prisoners dilemma is more or less the pinnacle of cooperation. Multiple shots just make it easier to cooperate. If you don't care about single shot PM you may be sacrificing human lives. Strategy: "give him the paperclips if you think he'll save the lives if and only if he expects you to give him the paperclips and you think he will guess your decision correctly".
You are right, I used the word 'cooperation' in the informal sense of 'does not want to destroy me'. I fully admit that it is hard to formalize this concept, but if it says noncooperating and the game theoretic definition says cooperating, I prefer my definition. :) A possible problem I see with this game theoretic framework is that in real life, the agents themselves set up the situation where cooperation/defect occurs. As an example: the PM navigates humanity into a PD situation where our minimal payoff is 'all humans dead' and our maximal payoff is 'half of humanity dead', and then it cooperates. I bumped into a question when I tried to make sense of all this. I have looked up the definition of PM at the wiki. The entry is quite nicely written, but I couldn't find the answer to a very obvious question: How soon does the PM want to see results in its PMing project? There is no mention of time-based discounting. Can I assume that PMing is a very long-term project, where the PM has a set deadline, say, 10 billion years from now, and its actual utility function is the number of paperclips at the exact moment of the deadline?
Blah blah blah Chinese room you are not really sentient!
Sapient, the word is sapient. Just about every single animal is capable of sensing.
I think this way of posing the question contains a logical mistake. Values aren't always justified by other values. The factual statement "I have this value because evolution gave it to me" (i.e. because I'm human, or because I'm white) does not imply "I follow this value because it favors humans, or whites". Of course I'd like FAI to have my values, pretty much by definition of "my values". But my values have a term for other people, and Eliezer's values seem to be sufficiently inclusive that he thought up CEV.

Here's something interesting on gender relations in ancient Greece and Rome.

Why did ancient Greek writers think women were like children? Because they married children - the average woman had her first marriage between the ages of twelve and fifteen, and her husband would usually be in his thirties.

The reason ancient Greek writers thought women were like children is the same reason men in all cultures think women are like children: There are significant incentives to do so. Men who treat women as children reap very large rewards compared to those men who treat women as equals. EDIT: If someone thinks this is an invalid point, please explain in a reply. If the downvote(s) is just "I really dislike anyone believing what he's saying is true, even if a lot of evidence supports it" (regardless of whether or not evidence currently supports it) then please leave a comment stating that. EDIT 2: Supporting evidence or retraction will be posted tonight. EDIT 3: As I can find no peer-reviewed articles suggesting this phenomenon, I retract this statement.

This conversation has been hacked.

The parent comment points to an article presenting a hypothesis. The reply flatly drops an assertion which will predictably derail conversation away from any discussion of the article.

If you're going to make a comment like that, and if you prefix it with something along the lines of "The hypothesis in the article seems superfluous to me; men in all cultures treat women like children because...", and you point to sources for this claim, then I would confidently predict no downvotes will result.

(ETA: well, in this case the downvote is mine, which makes prediction a little too easy - but the point stands.)

Thanks! I won't be able to do the work required on this right now, but will later tonight.
Wow, that's a great link.
LW doesn't like to hear the truth about male/female sexual strategies; we like to have accurate maps here, but there's a big "censored" sign over the bit of the map that describes the evolutionary psychology of sexuality, practical dating advice, the burgeoning "pick-up" community and an assorted cloud of topics. Reasons for this censorship (and I agree to an extent) are that talking about these topics offends people and splits the community. LW is more useful, it has been argued, if we just don't talk about them.

The PUA community include people who come across as huge assholes, and that could be an alternative explanation of why people react negatively to the topics, by association. I'm thinking in particular of the blog "Roissy in DC", which is on OB's blogroll.

Offhand, it seems to me that thinking of all women as children entails thinking of some adults as children, which would be a map-territory mistake around the very important topic of personhood.

I did pick up some interesting tips from PUA writing, and I do think there can be valuable insight there if you can ignore the smell long enough to dig around (and wash your hands afterwards, epistemically speaking).

No relevant topics should be off-limits to a community of sincere inquiry. Relevance is the major reason why I wouldn't discuss the beauty of Ruby metaprogramming on LessWrong, and wouldn't discuss cryonics on a project management mailing list.

If discussions around topic X systematically tend to go off the rails, and topic X still appears relevant, then the conclusion is that the topic of "why does X cause us to go off the rails" should be adequately dealt with first, in lexical priority. That isn't censorship, it's dependency management.

But in reality, this topic is off-limits. Therefore LW is not a community of sincere inquiry, but nothing's perfect, and LW does a lot of good. Interesting. However, in this case, that discussion might get somewhat accusatory, and go off the rails itself.
Got that. I am suggesting that it is off-limits because this community isn't yet strong enough at the skills of collaborative truth-seeking. Past failures shouldn't be seen as eternal limitations; as the community grows, by acquiring new members, it may grow out of these failures. To make this concrete, the community seems to have a (relative) blind spot around things like pragmatics, as well as what I've called "myths of pure reason". One of the areas of improvement is in reasoning about feelings. I'm rather hopeful, given past contributions by (for instance) Alicorn and pjeby.
I don't think that is the reason for the problem. The community doesn't go off the rails and have to censor discussions about merely pragmatic issues. It is more that the community has a bias surrounding the concept of traditional, storybook-esque morality, roughly a notion of doing good that seems to have some moral realist heritage, a heavy tint of political correctness, and sees the world in black-and-white terms, rather than moral shades of grey. Facts that undermine this conception of goodness can't be countenanced, it seems. Robin Hanson, on the other hand, has no trouble posting about the sexuality/seduction cluster of topics. There seems to be a systematic difference between OB and LW along this "moral political correctness/moral constraints" dimension - Robin talks with enthusiasm about futures where humans have been replaced with Vile Offspring, and generally shuns any kind of talk about ethics. (EDITED, thanks to Morendil)
This kind of phrase seems designed to rile (some of) your readers. You will improve the quality of discourse substantially by understanding that and correcting for it. Unless, of course, your goal really is to rile readers rather than to improve quality of discourse.
There is truth to what you say but unfortunately you are letting your frustration become visible. That gives people the excuse to assign you lower status and freely ignore your insight. This does not get you what you want. This is perhaps one of the most important lessons to be learned on the topic of 'pragmatics'. Whether you approach the topic from works like Robert Greene []'s on Power [], War [] and Seduction [] or from the popular social skills based self help communities previously mentioned [], a universal lesson is that things aren't fair, bullshit is inevitable and getting indignant about the bullshit gets in the way of your pragmatic goals. There may be aspects of the morality here that is childlike or naive and I would be interested in your analysis of the subject since you clearly have given it some thought. But if you are reckless and throw out 'like theists' references without thought your contribution will get downvoted to oblivion and I will not get to hear what you have to say. Around here that more or less invokes the 'nazi' rule. Edit: No longer relevant.
LOL... indeed. I am not sure that I am actually, in far mode, so interested in correcting this particular LW bias. In near mode, SOMEONE IS WRONG ON THE INTERNET bias kicks in. It seems like it'll be an uphill struggle that neither I nor existential risk mitigation will benefit from. A morally naive LW is actually good for X-risks, because that particular mistake (the mistake of thinking in terms of black-and-white morality and Good and Evil) will probably make people more "in the mood" for selfless acts of charity.
I think I agree. If Eliezer didn't have us all convinced that he is naive in that sense we would probably have to kill him before he casts his spell of ultimate power. (cough The AI Box demonstrations were just warm ups...)
Robin can do what he likes on his own blog without direct consequences within the blog environment. He also filters which comments he allows to be posted. I guess what I am saying is that it isn't useful to compare OB and LW on this dimension because the community vs individual distinction is far more important than the topic clustering.
I may not have been clear: I meant pragmatics in this sense [], roughly "how we do things with words". I'd also include things like denotation vs connotation in that category. Your comment on "pragmatic issues" suggests you may have understood another sense.
oh, ok. Linguistic pragmatics. That's a more fruitful idea.
Curiously, there is ambiguity there and both meanings seem to apply.
The article suggests a direct counter-example: by having high standards, the men forfeit the labor of the women in things like 'help[ing] with finance and political advice'. Much like the standard libertarian argument against discrimination: racists narrow their preferences, raising the cost of labor, and putting themselves at a competitive disadvantage. Men may as a group have incentive to keep women down, but this is a prisoner's dilemma.
4Paul Crowley13y
Why do so many people here believe that? It strongly contradicts my experience.

Your experience is atypical because you're atypical.

2Paul Crowley13y
Man, I've barely looked at that page since I wrote it four years ago. I live with Jess now, across the road from the other two. I can heartily recommend my brand of atypicality :-)
Good answer. I keep privately asking the same question about these sorts of things, and getting the same answer from others.
What do you mean by "many people here believe that"? Believe what? And what tells you they do believe it?
People have great difficulty verbalizing their perceptions of and beliefs about social interactions. It is not obvious to me that you two have different beliefs or experiences. More likely, you do, but it would probably take a lot of work to identify and communicate those differences.
I guess because our experience contradicts your experience.
Is that true? What are the incentives and rewards? Are there circumstances under which this is a bad idea - for example, do relative ages or relative social position matter? (For example, what if the woman in question is your mother, teacher/professor, employer, or some other authority figure with power over you?) Are there also incentives for men to treat other men as children, or for women to treat men or other women as children?
I wonder if adults treat children like children merely because of the benefits they reap by doing so.
Sometimes that's definitely the case. At other times it really does appear to be for real and concrete neutral reasons.
I'm pretty sure he's trying to say basically the same thing as this OB post [] (specifically the part from "Suppose that middle-class American men are told..." on).
Interesting read, thanks.
Plus women have more juvenile morphology compared to men. Women are shorter, smaller, less muscled, beardless, have higher voices, more fatty tissue, etc. The Greeks and Romans seemed to rely on surface analogies for reasoning.

Could someone discuss the pluses and minuses of ALCOR vs Cryonics Institute.

I think Eliezer mentioned that he is with CI because he is young. My reading of the websites seem to indicate that CI leaves a lot of work to be potentially done by loved ones or local medical professionals who might not be in the best state of mind or see fit to co-operate with a cryonics contract.


It's not at all obvious to me how to comparison-shop for cryonics. The websites are good as far as they go, but CI's in particular is tricky to navigate, funding with life insurance messes with my estimation of costs, and there doesn't seem to be a convenient chart saying "if you're this old and this healthy and this solvent and your family members are this opposed to cryopreservation, go with this plan from this org".
Alcor is better. CI is cheaper and probably good enough.
"Probably good enough" doesn't engender a lot of confidence. It would seem a tragedy to go through all of this and then not be reanimated because you carelessly chose the wrong org. On the other hand spending too much time trying to pick the right org does seem like raw material for cryocrastination. Does anyone have thoughts / links on whole body vitrification? ALCOR claims that this is less effective than going neuro, but CI doesn't seem to offer neuro option anymore.
0Paul Crowley13y
Disclaimer: I have no relevant expertise. That said, FWIW I suspect that whole-body people will be brought back first: * if through bodily reanimation, because repair of the whole body will be easier than replacement of the body given only the severed head * if through scanning/WBE, because it will be possible to scan their spinal columns as well as their brains and it will be easier to build them virtual bodies using their real bodies as a basis. Though CI don't offer a neuro option, their focus (obviously) is preserving the information in the brain.
Is Alcor in fact that much better than CI (plus SA, that is)?
Alcor both stores your body and provides for bedside "standby" service to immediately begin cooling. With CI, it's a good idea to contract a third party to perform that service, and SA is the recommended company to perform that service. []
It depends on how you define that much better, but probably not. The only concrete thing I know of is that Alcor saves and invests more money per suspendee.
2Eliezer Yudkowsky13y
I'd guess CI + SA > Alcor > CI.
I didn't know you thought CI + SA was actually better than Alcor regardless of cost. Have you said that in more words elsewhere on this site?

The Believable Bible

This post arose when I was pondering the Bible and how easy it is to justify. In the process of writing it, I think I've answered the question for myself. Here it is anyway, for the sake of discussion.

Suppose that there's a world very much like this one, except that it doesn't have the religions we know. Instead, there's a book, titled The Omega-Delta Project, that has been around in its current form for hundreds of years. This is known because a hundreds-of-years-old copy of it happens to exist; it has been carefully and precisely compared to other copies of the book, and they're all identical. It would be unreasonable, given the evidence, to suspect that it had been changed recently. This book is notable because it happens to be very well-written and interesting, and scholars agree it's much better than anything Shakespeare ever wrote.

This book also happens to contain 2,000 prophecies. 500 of them are very precise predictions of things that will happen in the year 2011; none of these prophecies could possibly be self-fulfilling, because they're all things that the human race could not bring about voluntarily (e.g. the discovery of a particular artifact, or the... (read more)

7Eliezer Yudkowsky13y
Pretty darned high, because at this point we already know that the world doesn't work the way we think it did.
So it sounds like even though there are 2,000 separate prophecies, the probability of every prophecy coming true is much greater than 2^(-2000).
Maybe you just need to explain more but I don't see that.
Let P(2,000) be the probability that all 2,000 prophecies come true, and P(500) be the probability that the initial 500 all come true. Suppose P(2,000) = 2^(-2000) and P(500) = 2^(-500). We know that P(500|2,000) = 1, so P(2,000|500) = P(2,000)*P(500|2,000)/P(500) = 2^(-2000)*1/2^(-500) = 2^(-1500). A probability of 2^(-1500) is not pretty darned high, so either P(2,000) is much greater than we supposed, or P(500) is much lower than we supposed. The latter is counterintuitive; one wouldn't expect the Believable Bible's existence to be strong evidence against the first 500 prophecies.
And this doesn't depend on prophecies in particular. Any claims made by the religion will do. For example, the same sort of argument would show that according to our subjective probabilities, all the various claims of a religion should be tightly intertwined. Suppose (admittedly an extremely difficult supposition) we discovered it to be a fact that 75 million years ago, an alien named Xeno brought billions of his fellow aliens to earth and killed them with hydrogen bombs. Our subjective probability that Scientology is a true religion would immediately jump (relatively) high. So one's prior for the truth of Scientology can't be anywhere near as low as one would think if one simply assigned an exponentially low probability based on the complexity of the religion. Likewise, for very similar reasons, komponisto's claim elsewhere that Christianity is less likely to be true than that a statue would move its hand by quantum mechanical chance events, is simply ridiculous.
If nobody had ever proposed Scientology, though, learning Xenu existed wouldn't increase our probabilities for most other claims that happen to be Scientological. So it seems to me that our prior can be that low (to the extent that Scientological claims are naturally independent of each other), but our posterior conditioning on Scientology having been proposed can't.
Right, because that "Scientology is proposed" has itself an extremely low prior, namely in proportion to the complexity of the claim.
In proportion to the complexity of the claim given that humans exist, which is much lower (=> higher prior) than its complexity in a simple encoding, since Scientology is the sort of thing that a human would be likely to propose.
The prior for "Scientology is proposed" is higher than the simple complexity prior of the claim, to the (considerable) extent that Scientology is the sort of thing a human would make up.
You've got it a little backward, I think. The fact that someone makes a particular set of prophesies does not make those things more likely to occur. In fact, the chances of the whole thing happening... the events prophesied and the prophesies themselves is much lower than one or the other happening by themselves. This means that if some of the prophesies start coming true the probability the other prophesies come true goes up pretty fast. But predicted magic is even less likely than magic.
Use \* to get stars * instead of italics.
Oops! It seems I assumed everything would come out right instead of checking after I posted.
Edit: Yeah, I was being dumb.
Where A = "events occur" and B = "events are predicted", you're saying P(A and B) < P(A). Warrigal is saying it would be counterintuitive if P(A|B) < P(B).
Where A = "events occur" and B= "events are prophesied" and C = "the events prophesied come true" I am saying that when the events in A= the events in B, P(A|B) < P(B) or P(A) because A ^ B entails C.
You're talking about P(A and B). Warrigal is talking about P(A|B).
But not necessarily over .99, since the prophecies could have been altered by another author sometime before the beginning of modern records.
Could be simple time travel, though. AFAICT time travel isn't per se incompatible with the way we think the world works. Not to the degree sufficiently fantastic prophecies might be at least.
If someone just observed events in 2011 and planted a book describing them in 1200, the 2011 resulting from the history where the book existed would be different from the 2011 he observed.
4Paul Crowley13y
Depends if it's type one time travel []. Fictional examples: Twelve Monkeys, The Hundred Light-Year Diary.
I think the important bit here is that even if you could just "play time backwards" and watch again, there's no reason to think you'd end up in the same Everett branch the next time around.
Insofar as I understand that page, that would mean that the world worked even less the way we thought it did.
Makes perfect sense to me if you assume a single time-line. (This might be a big assumption, but probably less big than the truth of sufficiently strange prophecies.) You can think of this time line as having stabilized after a very long sequence of attempts at backward time travel under slightly different conditions. Any attempt at backward time travel that changes its initial conditions means a different or no attempt at time travel happens instead. Eventually you end with a time-line where all attempts at backward time travel exactly reproduce their initial conditions. We know that we live in that stabilized time-line because we exist (though the details of this timeline depend on how people who don't exist, but would have thought they exist for the same reasons we think we exist, would have acted, had they existed).
By the way, that sort of time-travel gives rise to Newcomb-like problems: Suppose you have access to a time-machine and want to cheat on a really important exam (or make a fortune on the stock marked or save the world or whatever. The cheating example is the simplest). You decide to send yourself at a particular time a list with the questions after taking the exam. If you don't find the list at the time you decided you know that somehow your attempt at sending the list failed (you changed your mind, the machine exploded in a spectacular fashion, you were caught attempting to send the list ...). But if you now change your mind and don't try to send the list there never was any possibility of receiving the list in the first place! The only way to get the list if for you to try to send the list even if you already know you will fail, so that's what you have to do if you really want to cheat. And if you really would do that, and only then, you will probably get the list at the specified time and never have to do it without knowing you succeed, but only if your pre-commitment is strong enough to even do it in the face of failure. And if you would send yourself back useful information at other times even without having either received the information yourself or pre-commited to sending that particular information you will probably receive that sort of information.
Why was this post voted back down to 0 after having been at 2? Newcomb-like problems are on-topic for this site and I would think having examples of such problems in a scenario not specifically constructed for them is a good thing? If it was because time travel is off topic wouldn't the more prudent thing have been voting down the parent? The same if the time travel mechanics are considered incoherent (though I'd be really interested in learning why?) . If you think this post doesn't actually describe anything Newcomb-like I would like to know why. Maybe I misunderstood the point of earlier examples here, or maybe I didn't explain things sufficiently? Or is it just that the post was written badly? I'm not really happy with it, but I don't see how I could have made it much clearer.
It's an interesting point. It actually came up in the most recent Artemis Fowl novel, when he managed to 'precommit' himself out of a locked trunk in a car. :)
Anyone who can travel through time can mount a pretty impressive apocalypse and announce whatever it is about the nature of reality he cares to. He might even be telling the truth.
For the two examples of the mundane prophecies that you gave it seems possible some on-going conspiracy could have made them true... but it sounds like you're trying to rule that out.
I understood those to be negative examples, in that the actual prophecies don't share that characteristic with those examples.
I did mean those to be positive examples. There's no way we can guarantee that we'll discover an ancient Greek goblet that says "I love this goblet!" on March 22, 2011. There's also no way we can guarantee that a woman born on October 15, 1985 at 5 in the morning in room 203 of a certain hospital will have a baby weighing 8 pounds and 6 ounces on January 8, 2011 at 6 in the afternoon in room 117 of a certain other hospital.
That's not clear to me, but I acknowledge that it doesn't affect the original question.

The FBI released a bunch of docs about the anthrax letter investigation today. I started reading the summary since I was curious about codes used in the letters. All of a sudden on page 61 I see:

c. Godel, Escher, Bach: the book that Dr. Ivins did not want investigators to find

The next couple of pages talk about GEB and relate some parts of it to the code. It's really weird to see literary analysis of GEB in the middle of an investigation on anthrax attacks.

When new people show up at LW, they are often told to "read the sequences." While Eliezer's writings underpin most of what we talk about, 600 fairly long articles make heavy reading. Might it be advisable that we set up guided tours to the sequences? Do we have enough new visitors that we could get someone to collect all of the newbies once a month (or whatever) and guide them through the backlog, answer questions, etc?

Most articles link to those preceeding it, but it would be very helpful to have links to those articles that follow.
One example: The Thing That I Protect []. If that makes you want to know what the "last thing" is, you have to click Next no less than ten times on Articles tagged ai [] to find out. Another is "More on this tomorrow" in Resist the Happy Death Spiral [].
I found this [] (scroll down for the majority of articles) graph of all links between Eliezer's articles a while ago, it could be be helpful. And its generally interesting to see all the interrelations.
Yes- it's very natural for the ongoing community progression of LW, but not great for archiving; we're pulling up the latter after we've climbed it.
I'll edit this post to add if I want to add further examples. * One Life Against the World []: "I will post later on why this tends to be so.".
That's not a bad idea. How about just a third monthly thread? To be created when a genuinely curious newcomer is asking good, but basic questions. You do not want to distract from a thread but at the same time you may be willing to spend time on educational discussion.
I approve. This may also spawn new ways of explaining things.
Or create (or does one exist) some thread(s) that would be a standard place for basic questions. Having somewhere always open might be useful too.
Yes, I am working my way through the sequences now. Hearing these ideas makes one want to comment but so frequently its only a day or two before I read something that renders my previous thoughts utterly stupid. It would be nice to have a "read this and you won't be a total moron on subject X" guide. Also, it would be good to encourage the readings about Eliezer Intellectual Journey. Though its at the bottom of the sequence page I used it a "rest reading" between the harder sequences. It did a lot to convince me that I wasn't inherently stupid. Knowing that Eliezer has held foolish beliefs in the past is helpful.
Arguably, as seminal as the sequences are treated, why are the "newbies" the only ones who should be (re)reading them?
As a newcomer, I would find this tremendously useful. I clicked through the wiki links on noteworthy articles, but often find there are a lot of assumptions or previously discussed things that go mentioned but unexplained. Perhaps this would help.

I'm taking a software-enforced three-month hiatus from Less Wrong effective immediately. I can be reached at zackmdavis ATT yahoo fullstahp kahm. I thought it might be polite to post this note in Open Thread, but maybe it's just obnoxious and self-important; please downvote if the latter is the case thx

Given how much time I've spent reading this site lately, doing something like that is probably a good idea. Therefore, I am now incorporating Less Wrong into the day-week-month rule, which is a personal policy that I use for intoxicants, videogames, and other potentially addictive activities - I designate one day of each week, one week of each month, and one month of each year in which to abstain entirely. Thus, from now on, I will not read or post on Less Wrong at all on Wednesdays, during the second week of any month, or during any September. (These values chosen by polyhedrical die rolls.)
I'm not going to be posting/reading so much for a while. I need to change my headspace. I'll probably try your method when I want to get back in.
Awesome. Less Wrong does seem to be an addictive activity. Wanting to keep up with recent comments is one factor in this, and I think I lose more time than I've estimated doing so. Disciplined abstention is actually a really good solution. I will implement something analogous. For the next 40 days, I will comment only on even days of the month. (I cannot commit to abstaining entirely because I don't have the will-power to enforce gray areas ... for example, can I refresh the page if it's already open? Can I work on my post drafts? Can I read another chapter of The Golden Braid? Etc.) Later edit: ooh! Parent upvoted for very useful link to LeechBlock.

I feel like the 20-something whose friends are all getting married and quiting drinking. This is lame. The party is just starting guys!

Yeah... and I'm going into withdrawal already. What if somebody comments about one of my favorite topics -- tomorrow?!? It's like deciding to diet. As soon as I decide to go on a diet I start feeling hungry. It doesn't make any difference how recently I've eaten. Heck, if I'm currently eating when I make this decision, I'll eat extra ... Totally counter-productive for me. Nevertheless.
Weird— without having read this, I just mentioned LeechBlock too [] and pointed out that I've been blocking myself from LW during weekdays (until 5). I guess all the cool kids are doing it too...
Rehab is for quitters.
Why does everyone like LeechBlock? pageaddict [] works pretty well and has a far less convoluted interface. EDIT: and now pageaddict seems to be completely unmaintained and even the domain is expired. Oh well.
It's possible that I shouldn't try to other-optimize [] here, but in the case of recent comments, I wonder if it'd be practical to make a folder on your computer where you save a copy of the latest-comments page when you see something interesting, telling yourself you'll look when you have more time. Or first retrieve all recent comments (with wget or cURL, or just right-clicking and saving), then turn on Leechblock to look at them, so you at least have an inconvenience barrier between writing a comment and posting it. On another site, I found that first writing comments without posting them and then saving threads without reading them helped me feel less anxious about missing things, although I've been backsliding recently. Share Your Anti-Akrasia Tricks [] might be useful to save and read offline, or print out if you want to go extreme. [Comment edited once.]
This is to confess that I cheated several times by reading the Google cache.
Turning the siteblocker back on (including the Google cache, thank you). Two months, possibly more. Love &c.
Tsk, tsk. You can block the Google cache [] too.
Great plugin. In case you have a linux dev (virtual) machine I also recommend: sudo iptables -A INPUT -d -j DROP It does wonders for productivity!
I'm disappointed, but if you think you have better things to do, I won't object.

Here's a question that I sure hope someone here knows the answer to:

What do you call it when someone, in an argument, tries to cast two different things as having equal standing, even though they are hardly even comparable? Very common example: in an atheism debate, the believer says "atheism takes just as much faith as religion does!"

It seems like there must be a word for this, but I can't think what it is. ??

False equivalence?
Aha! I think this one is closest to what I have in mind. Thanks. It's interesting to me that "false equivalence" doesn't seem to have nearly as much discussion around it (at least, based on a cursory google survey) as most of the other fallacies. I seem to see this used for rhetorical mischief all the time!
Fair and balanced reporting. []
This is a great example of a "pitch". I've added it just now to the database of pitches: []
0Eliezer Yudkowsky13y
Closest I know is "tu quoque".
That is pretty close. If I understand them right, I think the difference is: Tu Quoque: X is also guilty of Y, (therefore Z). False Equivalence: (X is also guilty of Y), therefore Z. where the parentheses indicate the major location of error.

Could anyone recommend an introductory or intermediate text on probability and statistics that takes a Bayesian approach from the ground up? All of the big ones I've looked at seem to take an orthodox frequentist approach, aside from being intolerably boring.

(All of the below is IIRC.) For a really basic introduction, there's Elementary Bayesian Statistics []. It's not worth the listed price (it has little value as a reference text), but if you can find it in a university library, it may be what you need. It describes only the de Finetti coherence justification; on the practical side, the problems all have algebraic solutions (it's all conjugate priors, for those familiar with that jargon) so there's nothing on numerical or Monte Carlo computations. Data Analysis: A Bayesian Approach [] is a slender and straighforward introduction to the Jaynesian [] approach. It describes only the Cox-Jaynes justification; on the practical side, it goes as far as computation of the log-posterior-density through a multivariate second-order Taylor approximation. It does not discuss Monte Carlo methods. Bayesian Data Analysis, 2nd ed. [] is my go-to reference text. It starts at intermediate and works its way up to early post-graduate. It describes justifications only briefly, in the first chapter; its focus is much more on "how" than "why" (at least, for philosophical "why", not methodological or statistical "why"). It covers practical numerical and Monte Carlo computations up to at least journeyman level.
I'm not intending to put this out as a satisfactory answer, but I found it with a quick search and would like to see what others think of it. Introduction to Bayesian Statistics by William M. Bolstad [] Good reviews on Amazon, and available from $46 + shipping... []
It's hard to say from the limited preview, which only goes up to chapter 3 -- the Bayesian stuff doesn't start until chapter 4. The first three chapters cover basic statistics material -- it looks okay to my cursory overview, but will be of limited interest to people looking for specifically Bayesian material. As to the rest of the book, the section headings look about right.
2Eliezer Yudkowsky13y
I second the question. "Elements of Statistical Learning" is Bayes-aware though not Bayesian, and quite good, but that's statistical learning which isn't the same thing at all.

Discussions of correctly calibrated cognition, e.g. tracking the predictions of pundits, successes of science, graphing one's own accuracy with tools like PredictionBook, and so on, tend to focus on positive prediction: being right about something we did predict.

Should we also count as a calibration issue the failure to predict something that, in retrospect, should have been not only predictable but predicted? (The proverbial example is "painting yourself into a corner".)

That issue could be captured if there were some obvious way to identify issues where predictions should be made in advance. If they fail to make predictions, they are being careless; if their predictions are incorrect, they are incorrect.
I think so, but it's important to identify the time at which it became predictable - for example, you could only predict that you were painting yourself into a corner just prior to when you made the last brushstroke that made the strip(s) of paint covering the exit path too wide to jump over. This seems hard. Also, you'd have to know what your utility function was going to be in the future to know that some event was even worth predicting. This seems hard, too.

More cryonics: my friend David Gerard has kicked off an expansion of the RationalWiki article on cryonics (which is strongly anti). The quality of argument is breathtakingly bad. It's not strong Bayesian evidence because it's pretty clear at this stage that if there were good arguments I hadn't found, an expert would be needed to give them, but it's not no evidence either.

I have not seen RationalWiki before. Why is it called Rational Wiki?

From :

RationalWiki is a community working together to explore and provide information about a range of topics centered around science, skepticism, and critical thinking. While RationalWiki uses software originally developed for Wikipedia it is important to realize that it is not trying to be an encyclopedia. Wikipedia has dominated the public understanding of the wiki concept for years, but wikis were originally developed as a much broader tool for any kind of collaborative content creation. In fact, RationalWiki is closer in design to original wikis than Wikipedia.

Our specific mission statement is to:

  1. Analyze and refute pseudoscience and the anti-science movement, ideas and people.
  2. Analyze and refute the full range of crank ideas - why do people believe stupid things?
  3. Develop essays, articles and discussions on authoritarianism, religious fundamentalism, and other social and political constructs

So it's inspired by Traditional Rationality.

A fine mission statement, but my impression from the pages I've looked at is of a bunch of nerds getting together to mock the woo. "Rationality" is their flag, not their method: "the scientific point of view means that our articles take the side of the scientific consensus on an issue."

Voted up, but calling them "nerds" in reply is equally ad-hominem, ya know. Let's just say that they don't seem to have the very high skill level required to distinguish good unusual beliefs from bad unusual beliefs, yet. (Nor even the realization that this is a hard problem, yet.)

Yes, they're pretty softcore by LessWrongian standards but places like this are where advanced rationalists are recruited from, so if someone is making a sincere effort in the direction of Traditional Rationality, it's worthwhile trying to avoid offending them when they make probability-theoretic errors. Even if they mock you first.

Also, one person on RationalWiki saying silly things is not a good reason to launch an aggressive counterattack on a whole wiki containing many potential recruits.

Yes, they're pretty softcore by LessWrongian standards but places like this are where advanced rationalists are recruited from, so if someone is making a sincere effort in the direction of Traditional Rationality, it's worthwhile trying to avoid offending them when they make probability-theoretic errors. Even if they mock you first.

I guess I should try harder to remember this, in the context of my rather discouraging recent foray into the Richard Dawkins Forums -- which, I admit, had me thinking twice about whether affiliation with "rational" causes was at all a useful indicator of actual receptivity to argument, and wondering whether there was much more point in visiting a place like that than a generic Internet forum. (My actual interlocutors were in fact probably hopeless, but maybe I could have done a favor to a few lurkers by not giving up so quickly.)

But, you know, it really is frustrating how little of the quality of a person (like Richard Dawkins, or, say, Paul Graham) or a cause (like increasing rationality, or improving science education) actually manages to rub off or trickle down onto the legions of Internet followers of said person or cause.

But, you know, it really is frustrating how little of the quality of a person (like Richard Dawkins, or, say, Paul Graham) or a cause (like increasing rationality, or improving science education) actually manages to rub off or trickle down onto the legions of Internet followers of said person or cause.

This is actually one of Niven's Laws: "There is no cause so right that one cannot find a fool following it."

You understand this is more or less exactly the problem that Less Wrong was designed to solve.

Is there any information on how the design was driven by the problem? For example, I see a karma system, a hierarchical discussion that lets me fold and unfold articles, and lots of articles by Eliezer. I've seen similar technical features elsewhere, such as Digg and SlashDot, so I'm confused about whether the claim is that this specific technology is solving the problem of having a ton of clueless followers, or the large number of articles from Eliezer, or something else.
not to detract, but does Richard Dawkins really posses such 'high quality'? IMO his arguments are good as a gateway for aspiring rationalists, not that far above the sanity water line [] that, or it might be a problem of forums in general ..

Dawkins is a very high-quality thinker, as his scientific writings reveal. The fact that he has also published "elementary" rationalist material in no way takes away from this.

He's way, way far above the level represented by the participants in his namesake forum.

(I'd give even odds that EY could persuade him to sign up for cryonics in an hour or less.)

(I'd give even odds that EY could persuade him to sign up for cryonics in an hour or less.)

Bloggingheads are exactly 60 minutes.

To be fair, I'd expect it to be a lot harder with an audience.
Exactly what I was thinking.
I was thinking: "Bloggingheads implies the participants believe they are within a few degrees of status of each other". It'd definitely be one worth a viewing or two!
Here's Dawkins on some non socially-reinforced views: AI [], psychometrics [], and quantum mechanics [] (in the last 2 minutes, saying MWI is slightly less weird than Copenhagen, but that the proliferation of branches is uneconomical).
8Paul Crowley13y
Obviously the most you could persuade him of would be that he should look into it.
you're absolutely right, I didn't consider his scientific writings, though my argument still weakly stands since I wasn't talking about that, he's a good scientist, but a rationalist of say Eliezer's level? I somehow doubt that. (my bias is that he hasn't gone beyond the 'debunking the gods' phase in his not specifically scientific writings, and here I'll admit I haven't read much of him.)

Read his scientific books, and listen to his lectures and conversations. Pay attention to the style of argumentation he uses, as contrasted with other writers on similar topics (e.g. Gould). What you will find is that beautiful combination of clarity, honesty, and -- importantly -- abstraction that is the hallmark of an advanced rationalist.

The "good scientist, but not good rationalist" type utterly fails to match him. Dawkins is not someone who compartmentalizes, or makes excuses for avoiding arguments. He also seems to have a very good intuitive understanding of probability theory -- even to the point of "getting" the issue of many-worlds.

I would indeed put him near Eliezer in terms of rationality skill-level.

Most of Dawkins' output predates the extreme rationality movement. Few scientists actually study rational thought - it seems as though the machine intelligence geeks and some of their psycholgist friends have gone some way beyond what is needed for everyday science.

Again, it's not just the fact that he does science; it's the way he does science.

Having skill as a rationalist is distinct from specializing in rationality as one's area of research. Dawkins' writings aren't on rational thought (for the most part); they're examples of rational thought.

I was actually considering writing a post about the term "Middle World []" - an excellent tool for capturing a large space of consistent weaknesses in human intuitions.
I was expecting him to write like the posts about rationality etc, but you make a good point. consequentially I was browsing the archives a while ago and found this [], now it is three ears old, but form the comments of Barkley_Rosser-mainly- it appears Gould didn't exactly "[undo] the last thirty years of progress in his depiction of the field he was criticizing" not that I want to revive that old thread.
Convincing Dawkins would be a great strategy for promoting cryonics... who else should the community focus on convincing?
Excusemewhat, the community, as in LW? We're a cryonics advocacy group now?
I used cryonics as example because komponisto used it before me. I intended my question to be more general. "If you're trying to market LW, or ideas commonly discussed here, then which celebrities and opinion-leaders should you focus on?"
Friends and family. They are the ones I care about most. (And, most likely, those that others in the community care about most too. At least the friends part. Family is less certain but more significant.)
Sure, convince those you love. I was asking who you should try to convince if your goal is convincing someone who will themselves convince a lot of other people.
Wait, you have a model which says it should? You don't learn from a person merely by associating with them. And: I would bet a fair bit that this is the source of your frustration, right there: scale. You can learn from a person by directly interacting with them, and sometimes by interacting with people who learned from them. Beyond that, it seems to me that you get "dilution effects", kicking in as soon as you grow faster than some critical pace at which newcomers have enough time to acculturate and turn into teachers. Communities of inquiry tend to be victims of their own success. The smarter communities recognize this, anticipate the consequences, and adjust their design around them.
Bad ones certainly seem to. Perhaps the high quality person at least leaves less room for the negative influences?
Interesting. Hom many places have you brought this issue up? Is there any forum which has responded rationally? What seem to be the controlling biases?
LW is thus far the only forum on which I have personally initiated discussion of this topic; but obviously I've followed discussions about it in numerous other places. You're on it. I mean, there are plenty of instances elsewhere of people getting the correct answer. But basically what you get is either selection bias (the forum itself takes a position, and people are there because they already agree) or the type of noisy mess we see at RDF. To date, LW is the only place I know of where an a priori neutral community has considered the question and then decisively inclined in the right direction. In the case of RDF, I suspect compartmentalization is at work: this topic isn't mentally filed under "rationality", and there's no obvious cached answer or team to cheer for. So people there revert to the same ordinary, not-especially-careful default modes of thinking used by the rest of humanity, which is why the discussion there looks just like the discussions everywhere else. It's noteworthy that my references and analogies to concepts and arguments discussed by Dawkins himself had no effect; apparently, we were just in a sort of separate magisterium. Particularly telling was this quote: Now on the face of it this seems utterly dishonest: I hardly think this fellow would actually be tempted to convert to theism upon hearing the news that eight Perugians had been convinced of God's existence. But I suspect he's actually just trying to express the separation that apparently exists in his mind between the kind of reasoning that applies to questions about God and the kind of reasoning that applies to questions about a criminal case.
Technical nitpick on the use of 'a priori' in the context. Subject to possible contradiction if I have missed a nuance in the meaning in the statistics context []). I would have just gone with 'previously'.
(As an extreme example, a few weeks idly checking out RationalWiki led me to the quote at the top of this page [] and only a few months after that I was at SIAI.)
I only just noticed this. Good Lord. (I put that quote there, so you're my fault.)
Point taken.
The realization that, as a human, you have something called an irrationality problem is both important and rare.

Just saw this over at Not Exactly Rocket Science:

Quick summary: They asked a bunch of people to give a 4-minute presentation, had people judging, and told the presenter how long it would be before they heard their assessment. Anticipating quicker feedback resulted in better performance, but predictions of worse performance, and anticipating slower feedback had the reverse effect.

The prosecutor's fallacy is aptly named:

Barlow and her fellow counsel, Kwixuan Maloof, were barred from mentioning that Puckett had been identified through a cold hit and from introducing the statistic on the one-in-three likelihood of a coincidental database match in his case—a figure the judge dismissed as "essentially irrelevant."

One thing that I got from the Sequences is that you can't just not assign a probability to an event - I think of this as a core insight of Bayesian rationality. I seem to remember an article in the Sequences about this where Eliezer describes a conversation in which he is challenged to assign a probability to the number of leaves on a particular tree, or the surname of the person walking past the window. But I can't find this article now - can anyone point me to it? Thanks!

6Vladimir_Nesov13y [] []
2Paul Crowley13y
That's exactly it - thanks!
This may be related to the recent post Study: Making decisions makes you tired []. It seems plausible that we don't assign probabilities to events until we have to, in order to make a decision, and that's why making decisions is tiring.

How do people decide what comments to upvote? I see two kinds of possible strategies:

  1. Use my approval level of the comment to decide how to vote (up, down or neutral). Ignore other people's votes on this comment.
  2. Use my approval level to decide what total voting score to give the comment. Vote up or down as needed to move towards that target.

My own initial approach belonged to the first class. However, looking at votes on my own comments, I get the impression most people use the second approach. I haven't checked this with enough data to be really certa... (read more)

I self-identify as using the first one, with a caveat. The second is obviously awful for communicating any sort of information given that only the sum of votes is displayed rather than total up and total down. The second is order dependent and often means you'll want to change your vote later based purely on what others think of the post. My "strategy" is to vote up and down based on whether I'd have wanted others with more insight than me to vote to bring my attention to or away from a comment, unless I feel I have special insight, in which case it's based on whether I want to bring others' attention to or away from a comment. This is because I see the goal of the voting process that readers' independent opinions on how much a comment is worth readers' attention be aggregated and used to bring readers' attention to or away from a comment. As a side effect, the author of a comment can use the aggregated score to determine whether her readers felt the comment was worth their collective attention. Furthermore since each reader's input comes in distinct chunks of exactly -1, 0, or +1, it's wildly unlikely that voting very often results in the best aggregation: instead I leave a comment alone unless I feel it was(is) significantly worth or not worth my(your) attention. The caveat: there is a selection effect in which comments I vote on, since my attention will be drawn away from comments with very negative karma. There is also undoubtedly an unconscious bias away from voting up a comment with very high karma: since I perceive the goal to be to shift attention, once a comment has very high karma I know it's going to attract attention so my upvote is in fact worth fewer attention-shift units. But I haven't yet consciously noticed that kick in until about +10 or so.
At home I use the Anti-Kibitzer, which enforces 1. I've been on vacation for a couple days and noticed the temptation to use 2. Gave in on one occasion, I'm afraid. On balance I'll stick to 1, as 2 seems too vulnerable to information cascades.
It is worth noting that people have explicitly claimed to be following strategy 2 here []. Edit: This is far from the only example; just the one I found by searching for "upvoted to". It would be also interesting to check the difference between comments on quotes threads and comments on substantive posts - at least one person has proposed that quotations are disproportionately subject to strategy 1 voting over strategy 2.

Am I/are we assholes? I posted a link to the frequentist stats case study to reddit:

The only commenter seems to have come to a conclusion from us that Bayesians are assholes.

Is it just that commenter, or are we really that obnoxious? (now that I think about it, I think I've actually seen someone else note something similar about Bayesians.) So... have we gone into happy death spiral "we get bonus points for acting extra obnoxious about those that are not us"?

It is common practice, when debating an issue with someone, to cite examples.

Has anyone else ever noticed how your entire argument can be undermined by stating a single example or fact which is does not stand up to scrutiny, even though your argument may be valid and all other examples robust?

Is this a common phenomenon? Does it have a name? What is the thought process that underlies it and what can you do to rescue your position once this has occurred?

It takes effort to evaluate examples. Revealing that one example is bad raises the possibility that others are bad as well, because the methods for choosing examples are correlated with the examples chosen. The two obvious reasons for a bad example are: 1. You missed that this was a bad example, so why should I trust your interpretation or understanding of your other examples? 2. You know this is a bad example, and included it anyway, so why should I trust any of your other examples?

The third horn of the anthropic trilemma is to deny that there is any meaningful sense whatsoever in which you can anticipate being yourself in five seconds, rather than Britney Spears; to deny that selfishness is coherently possible; to assert that you can hurl yourself off a cliff without fear, because whoever hits the ground will be another person not particularly connected to you by any such ridiculous thing as a "thread of subjective experience".

A question of rationality. Eliezer, I have... (read more)

From EY's post: Suppose I build a (conscious) brain in hardware using today's technology. It uses a very low current density, to avoid electromigration. Suppose I build two of them, and we agree that both of them experience consciousness. Then I learn a technique for treating the wafers to minimize electromigration. I create a new copy of the brain, the same as the first copy, only using twice the current, and hence being implemented by a flow of twice as many electrons. As far as the circuits and the electrons travelling them are concerned, running it is very much like running the original 2 brains physically right next to each other in space. So, does the new high-current brain have twice as much conscious experience?
I'm not as versed in this trilemma as I'd like to be, so I'm not sure whether that final question is rhetorical or not, though I suspect that it is. So mostly for my own benefit: While there's no denying that subjective experience is 'a thing', I see no reason to make that abstraction obey rules like multiplication. The aeroplane exists at a number of levels of abstraction above the atoms it's composed of, but we still find it a useful abstraction. The 'subjective experiencer' is many, many levels higher again, which is why we find it so difficult to talk about. Twice as many atoms doesn't make twice as much aeroplane, the very concept is nonsense. Why would we think any differently about the conscious self? My response to the 'trilemma' is as it was when I first read the post - any sensible answer isn't going to look like any of those three, it's going to require rewinding back past the 'subjective experience' concept and doing some serious reduction work. 'Is there twice as much experience?' and 'are you the same person?' just smell like such wrong questions to me. Anyone else? Nick, will have a look at that Bostrom piece, cheers.
Nick Bostrom's "Quantity of Experience" [] discusses similar issues. His model would, I think, answer "no", since the structure of counterfactual dependences is unchanged.

I'm new to Less Wrong. I have some questions I was hoping you might help me with. You could direct me to posts on these topics if you have them. (1) To which specific organizations should Bayesian utilitarians give their money? (2) How should Bayesian utilitarians invest their money while they're making up their minds about where to give their money? (2a) If your answer is "in an index fund", which and why?

This should help. [] In general, the best charities are SIAI [], SENS [] and FHI [].
I disagree. I recommend the top rated charities on [], specifically the Stop TB Partnership []. (They also have a nice blog [].)
On the other hand, I am willing to donate to SIAI out of my "donate to webcomics" mental account instead of my "save lives" mental account. ;) Regardless of whether or not he ever solves the Friendly AI problem, Eliezer's writing, on this blog and elsewhere, has given me enough of what might pejoratively be called "entertainment value" for me to want to pay him to keep doing it.
Why don't SIAI and FHI get evaluated by GiveWell? Maybe there would be some confusion regarding their less direct ways of helping people but I'd at least like some information about their effectiveness at what they claim to do. Or maybe that information is out there already. Anyone?
They are weird.
Basically, yes, with "They" referring to SIAI and FHI.
That's how I interpreted it, but I see the ambiguity now that you mention it. It doesn't help that the two statements are basically equivalent if you use "weird" as a relative term [].
GiveWell is a pretty small organization, and they haven't yet devoted any resources to evaluating research-based charities - they're looking for charities that can prove that they're providing benefits today, and lots of research ends up leading nowhere. How many increments of $1,000 - the amount it takes to cure an otherwise fatal case of tuberculosis - have been spent on medical research that amounted to nothing? For the record, I agree that SIAI is doing important work that must be done someday, but I don't expect to see AGI in my lifetime; there's no particular urgency involved. If Eliezer and co. found themselves transported back in time to 1890, would they still say that solving the Friendly AI problem is the most important thing they could be doing, given that the first microprocessor was produced in 1971? I'd tell them that the first thing they need to do is to go "discover" that DDT (first synthesized in 1874) kills insects and show the world how it can be used to kill disease vectors such as mosquitoes; DDT is probably the single man-made chemical that, to date, has saved more human lives than any other.

If Eliezer and co. found themselves transported back in time to 1890, would they still say that solving the Friendly AI problem is the most important thing they could be doing, given that the first microprocessor was produced in 1971?

In 1890, the most important thing to do is still FAI research. The best case scenario is that we already had invented the math for FAI before the first vacuum tube, let alone microchip. Existential risk reduction is the single highest utility thing around. Sure, trying to get nukes never made or made by someone capable of creating an effective singleton is important, but FAI is way more so.

Well, what if he were sent back to Ancient Greece (and magically acquired the ability to speak Greek)? Even if he got all the math perfectly right, who would care? Or even understand it?
He would then spend the rest of his life ensuring that it is preserved. If necessary he would go around hunting for obscure caves with a chisel in hand. Depending, of course, on how much he cares about influencing the future of the universe as opposed to other less abstract goals.
Yes, who today cares what any Greek mathematician had to say... Now you're just moving the goal posts.
Sorry. :( Anyway, I have much more confidence that Eliezer and future generations of Friendly AI researchers will succeed in making sure that nobody turns on an AGI that isn't Friendly than in Eliezer and his disciples solving both the AGI and Friendly AI problems in his own lifetime. Friendly AI is a problem that needs to be solved in the future, but, barring something like a Peak Oil-induced collapse of civilization to pre-1920 levels, the future will be a lot better at solving these problems than the present is - and we can leave it to them to worry about. After all, the present is certainly better positioned to solve problems like epidemic disease and global warming than the past was.
Would you consider SENS a viable alternative to SIAI? Or do you think ending aging is also impossible/something to be put off?
Actually, I would; I've donated a small amount of money already. Investing in anti-aging research won't pay off for at least thirty years - that's the turnaround time of medical research from breakthrough to useable treatment - but it's a lot less of a pie-in-the-sky concern. (Although as long as people are dying for want of $1,000 TB medication, it still might be more cost effective to save those lives than to extend the lives of relatively rich people in developed countries.)
My guess is that SENS is more cost effective, but I haven't done the calculating. Does anyone have access to those sorts of figures? Ball parking: $1000 buys you 45 extra person-years. $10 billion buys you 30 extra person-years for a billion people. Of course that depends on how much you agree with the figures given by de Grey.
I don't think he is if the point is to establish that "lack of FAI could at some point lead to Earth's destruction" isn't a unconditionally applicable argument.
That's an easy prediction for you to make. ;)
Well, I don't expect that my brother will see AGI in his lifetime, either.
I am curious: are you very old or suffering from a fatal disease? I am 25 and healthy, so "lifetime" probably means something different to me...
It's an ironic remark about my depression. I'm 27 and physically healthy.
That would make my winking outright cruel! No, I'm referring to the general problem of betting against the success of the person with whom you are making the bet. In CronoDAS's case the threshold for a self-sabotage outcome is somewhat reduced by his expressed suicidal inclinations.
See my reply to Lucas. Edit: Also, I'm sympathetic to your skepticism re: SIAI as the best charity.
I think it is precisely to that effect that this paper [] is aimed. Lets see when the paper comes out, lets see how persuasive it is. Edited for formatting
I think I need to clarify here. I am personally convinced (am a one-time donor myself), but the optimal charity argument in favour of Friendly AI research and development (which will be fully developed in this paper) is something I can use with my friends. They are pretty much the practical type and will definitely respond to wanting more bang for their buck and where their marginal rupee of charity should go. There are inferential gaps. And when me, a known sci-fi fan presents the argument, I get all sorts of looks. If I have a peer reviewed paper to show them, that would work nicely in my favour.
Sounds like a good idea to me.
I believe that the answer is a combination of the fact that SIAI and FHI aren't on their list (of charities to evaluate), as well as the fact that their methodology is heavily dependent on quality of information, and actual evidence that the charity is working.
Sure. But if GiveWell isn't going to do it then someone should. Are their budgets public? So many people here are skeptical of regular charities what evidence is there that these charities are different?
I don't think they publish a full budget but there is a breakdown of what the current fundraising drive is for. []
Could you explain why? Do you believe that SIAI/FHI aren't accomplishing what they set out to do? Do you discount future lives? Something else?
I don't expect Eliezer and co. to succeed, if you define "success" as actually building a transhuman Friendly AI before Eliezer is either cryopreserved or suffers information-theoretic death. My "wild guess" at the earliest plausible date for AGI of any kind is 2100.
5Eliezer Yudkowsky13y
What do you think you know and how do you think you know it?
I'm guessing based on several factors: 1) The past failure of AGI research to deliver progress 2) The apparent difficulty of the problem. We don't know how to do it, and we don't know what we would need to know before we can know how to do it. Or, at least, I don't. 3) My impressions of the speed of scientific progress in general. For example, the time between "new discovery" and "marketable product" in medicine and biotechnology is about 30 years. 4) My impressions of the speed of progress in mathematics, in which important unsolved problems often stay unsolved for centuries. It took over 300 years to prove Fermat's Last Theorem, and the formal mathematics of computation is less than a century old; Alan Turing described the Turing Machine in 1937. 5) The difficulty of computer programming in general. People are bad at programming.
Do you also evaluate the chances of WBE as being vanishingly slim over the next century?
Actually, no, but I also expect that it'll be around for quite a while before running a whole brain emulation becomes cheaper than hiring a human engineer. I don't expect a particularly fast em transition; it took many years for portable telephones to go from something that cost thousands of dollars and went in your car to the cell phones that everyone uses today. The Singularity was created by Nikola Tesla and Thomas Edison, and ended some time around 1920. Get used to it. ;)
So you expect that WBE will become possible before cheap supercomputers?
You might like to quantify "cheap" and "super".
See reply to CronoDAS below.
Even at Moore's Law speeds, simulating 10^11 neurons, 10^11 glial cells, 10^15 synaptic connections, and concentrations of various neurotransmitters and other chemicals in real time or faster-than-real time is going to be expensive for a long time before it becomes cheap.
Not necessarily. If a human brain with no software tricks requires 10^20 CPS (a very high estimate), then (according to Kurzweil, take with grain of salt) the computational capacity will be there by ~2040. However, it's certainly possible that we don't get the software until 2050, at which point anyone with a couple hundred dollars can run one.
Depends on which details actually need to be simulated. I suspect that most intracellular activity can be neglected or replaced with some simple rules on when a cell divides, adds a synapse, etc.
For the record, this is something I don't have much confidence in - WBE requires a sufficiently detailed brain scan, computers of sufficient processing power to run the simulation, and enough knowledge of brains on the microscopic level to program a simulation and understand the output of the simulation. I do not know which will turn out to be the bottleneck in the process.
It looks like "enough knowledge of brains on the microscopic level to program a simulation" might be the limiting factor. In which case, we have a hardware overhang and an explosive em transition.
Most technological developments seem to go from "We don't know how to do this at all" to "We know how to do this, but actually doing it costs a fortune" to "We know how to do this at an affordable price." WBE could be an exception, though, and completely skip over the second stage.
I disagree, but we probably have different estimates as to just how effective DNA modification and/or intelligence enhancing drugs are going to be in the future. I don't think Eliezer is going to make all that big of a dent in the FAI problem until he becomes more intelligent, and it's hard to estimate how much faster that will make him. I think I can say that intelligence enhancement could turn an impossible problem into a possible problem. It also means that there will be many more people out there capable of making meaningful contributions to the FAI problem.

Someone once told me that the reason they don't read Less Wrong is that the articles and the comments don't match. The articles have one tone, and then the comments on that article have a completely different tone; it's like the article comes from one site and the comments come from another.

I find that to be a really weird reason not to read Less Wrong, and I have no idea what that person is talking about. Do you?

Someone once told me that the reason they don't read Less Wrong is that the articles and the comments don't match...I have no idea what that person is talking about. Do you?


Back in Overcoming Bias days, I constantly had the impression that the posts were of much higher quality than the comments. The way it typically worked, or so it seemed to me, was that Hanson or Yudkowsky (or occasionally another author) would write a beautifully clear post making a really nice point, and then the comments would be full of snarky, clacky, confused objections that a minute of thought really ought to have dispelled. There were obviously some wonderful exceptions to this, of course, but, by and large, that's how I remember feeling.

Curiously, though, I don't have this feeling with Less Wrong to anything like the same extent. I don't know whether this is because of the karma system, or just the fact that this feels more like a community environment (as opposed to the "Robin and Eliezer Show", as someone once dubbed OB), or what, but I think it has to be counted as a success story.


Oh! Maybe they were looking at the posts that were transplanted from Overcoming Bias and thinking those were representative of Less Wrong as a whole.

I think that the situation about the imported OB posts & comments should be somehow made clear to new readers. Several things there (no embedded replies, little karma spent, plenty of inactive users, different discusion tone) could be a source of confusion.
9Eliezer Yudkowsky13y
I hate to sound complimentary, but... I get the impression that the comments on LW are substantially higher-quality than the comments on OB. And that the comments on LW come from a smaller group of core readers as well, which is to some extent unfortunate. I wonder if it's the karma system or the registration requirement that does it?

Less Wrong, especially commenting on it, is ridiculously intimidating to outsiders. I've thought about this problem, and we need some sort of training grounds. Less Less Wrong or something. It's in my queue of top level posts to write.

So the answer to your question is the karma system.

What's so intimidating? You don't need much to post here, just a basic grounding in probability theory, decision theory, metaethics, philosophy of mind, philosophy of science, computer science, cognitive bias, evolutionary psychology, the theory of natural selection, artificial intelligence, existential risk, and quantum mechanics - oh, and of course to read a sequence of >600 3000+ word articles. So long as you can do that and you're happy with your every word being subject to the anonymous judgment of a fiercely intelligent community, you're good.

Sounds like a pretty good filter for generating intelligent discussion to me. Why would we want to lower the bar?

Being able to comment smartly and in a style that gets you upvoted doesn't really need any grounding in any of those subjects. I just crossed 1500 karma and only have basic grounding in Computer Science, Mathematics, and Philosophy.

When I started out, I hadn't read more than EY's Bayes' for Dummies, The Simple Truth, and one post on Newcomb's.

In my opinion, the following things will help you more than a degree in any of the subjects you mentioned:

  • Crave the truth
  • Accept Reality as the source of truth
  • Learn in small steps
  • Ask questions when you don't understand something
  • Test yourself for growth
  • Be willing to enter at low status
  • Be willing to lose karma by asking stupid questions
  • Ignore the idiots
Another factor: * Being willing to shut up about a subject when people vote it down. So far as I am aware, the chief reason non-spammers have been banned is for obnoxious evangelism for some unpopular idea. Many people have unpopular ideas but continue to be valued members (e.g. Mitchell_Porter).
0Paul Crowley13y
Useful data point, thanks. Have you made any more progress with the sequences since you last updated your wiki user info page []?
Yeah. I just updated it again. I didn't realize anyone was actually looking at it... :P Recently I burned out on the sequences and am taking a break to gorge myself on other subjects. I tend to absorb large scale topics in rotation. It helps me stay focused over longer distances and has an added benefit of making me reread stuff that didn't stick the first time through. The weekly study group will also help data retention. Other data points that may be relevant: I have participated in a lot of online discussions; I have moved cross country into a drastically different cultural zone; I married someone from a non-US culture; I have visited at least one distinct Central American culture. In addition, I grew up in a religious culture but personally learn more toward a scientific/realistic culture. All of these things help build awareness that what I say isn't what other people hear and vice versa. As evidence of this, my conversations here have much better transmission success than my posts. Once I get to talk to someone and hear them respond I can start the translation predictors. I am still learning how to do this before I hear the responses.
8Eliezer Yudkowsky13y
Not "and". "Or". If you don't already have it, then reading the sequences will give you a basic grounding in probability theory, decision theory, metaethics, philosophy of mind, philosophy of science, computer science, cognitive bias, evolutionary psychology, the theory of natural selection, artificial intelligence, existential risk, and quantum mechanics.

I actually think this is a little absurd. There is no where near enough on these topics in the sequences to give someone the background they need to participate comfortably here. Nearly everyone here as a lot of additional background knowledge. The sequences might be a decent enough guide for an autodidact to go off and learn more about a topic but there is no where near enough for most people.

The sequences are really kind of confusing... I tried linking people to Eliezer's quantum physics sequence on Reddit and it got modded highly, but one guy posted saying that he got scared off as soon as he saw complex numbers. I think it'll help once a professional edits the sequences into Eliezer's rationality book. []
Which people do we want? What do those people need? However strongly you catapult a plane from the flight deck, at some point it has to fly by itself.

Without new blood communities stagnate. The risk of group think is higher and assumptions are more likely to go unchecked. An extremely homogeneous group such as this one likely has major blind spots which we can help remedy by adding members with different kinds of experiences. I would be shocked if a bunch of white male, likely autism spectrum, CS and hard science types didn't have blind spots. This can be corrected by informing our discussions with a more diverse set of experiences. Also, more diverse backgrounds means more domains we can comfortably apply rationality to.

I also think the world would be a better place if this rationality thing caught on. It is probably impossible (not to mention undesirable) to lower the entry barrier so that everyone can get in. But I think we could lower the barrier so that it is reasonable to think that 80-85+ percentile IQ, youngish, non-religious types could make sense of things. Rationality could benefit them and they being more rational could benefit the world.

Now we don't want to be swamped with newbies and just end up rehashing everything over and over. But we're hardly in any danger of that happening. I could be wrong but I suspect alm... (read more)

Data point to the contrary here. On top of being a data point, I'm also a person, which is convenient: you can ask a person questions. ;)
Actually, I'm something of a partial data point against that as well. I did come here with the split from OB but I was just a causal reader, had only been there a few weeks and never commented. I did go back and look at some of your early comments and my initial reaction is that you seem unusually well read and instinctively rational even for this crowd. In fact, I wonder if you should be asking me questions about how to make Less Wrong more amenable to people with limited background knowledge. You may regret this. I'm a very curious person. To what extent was it obvious upon coming here that Less Wrong had a kind of affinity with computer science and programming? What effect did this affinity have on your interest? How much of your interest in participating was driven by Eliezer's writings in particular compared to the community in general. Should the barrier to participation be lowered? If so how would you do it? What would have gotten you up to speed with everyone else faster? What would have made it easier? To what extent did you/do you now associate yourself with transhumanism? Did that factor into your interest in Less Wrong? I could probably keep going. One more for now: What questions should I be asking you that I haven't?
That's one question I really like. Originally learnt it from Jerry Weinberg as one of the "context free questions", a very useful item in my toolkit. What I'd ask people is "What motivated you to come here in the first place?" Where "here" takes a range of values - what motivated them to become readers; then to create a profile here and become commenters; then to start contributing. Not very. What "hooked" me first, as a huge Dennett fan, was the Zombies [] post, which I came across while browsing random links from my Twitter feed. That led me to the QM sequence, which made sense for me of things that hadn't made sense before, which motivated me to drill for more. That led me to the Bayes article. Parallel exploration turned up the FAI argument, which (I don't dare use the word "click" yet) made intuitive sense even though it hadn't crossed my mind before. It was only then that I made the connection with CS/programming - I had this fantasy of getting my colleagues to invite Eliezer to keynote at our conference. Interestingly enough, the response I got from musing about that on Twitter was (direct quote) "the singularity will definitely have personality disorders". Well, I'd taken note that there was such a thing as the LW community blog, and I kept an eye on it, but in parallel I started reading all of the back-content of LW, all the posts by Eliezer ported over from OB. I wanted to catch up before increasing my participation. So initially I pretty much ignored the community, which anyway I couldn't quite figure out. I wish someone had told me, quite plainly, what I was expected to do! Something along the lines of, "this is a rationality dojo, posts are where intermediate students show off their moves, comments are where beginners learn from intermediates and demonstrate their abilities, you will be given cues by people ahead of you when you are ready to move along the path reader->commenter->poster". Looking back, I
There is a wiki, though it sadly uses a different authentication system. Nonetheless, many users do have profile pages there.
I'm going to remember that one.
Indeed - the reason we don't say that explicitly is that it's unclear how much this is the case. However, if it were possible for Lw to become a "rationality dojo", I think most of us would leap on the opportunity.
There is some previous discussion [] which suggests that not everyone here would be happy to see LW as a "rationality dojo". The term "dojo" has favorable connotations for me, partly because one of my secret identity's modest claims to fame is as a co-originator of the "Coding Dojo", an attempt to bring a measure of sanity back to the IT industry's horrible pedagogy [] and hiring practices []. However these connotations might be biasing my thinking about whether using the "dojo" metaphor as a guide to direct the evolution of LW would be for good or ill on balance. How about starting a discussion at the top of the current Open Thread to ask people what they now think of applying the Dojo metaphor to LW?
I think I'm the only one on that thread who explicitly advised against starting a rationality dojo, and the other concerns were mostly whether it was possible.
Indeed. Eliezer's post itself, however, seemed mostly to caution against it, and perhaps what he took away from the subsequent discussion, after weighing the various contributions, was that it had too little to recommend it. At any rate, that I'm aware, the question wasn't raised again? Of course one issue is that it was never clarified what "it" might be, i.e. what would result from treating LW more explicitly as a "rationality dojo" (that would be different from what it is at present).
I believe purely ballistic transportation systems have been proposed at various times, actually.
I've long said that Truly Part of You [] is the article with, by far, the highest ratio of "Less Wrong philosophy content" to length. (Unfortunately, it doesn't seem to be listed in any sequence despite being a follow-up to two others.) Other than knowing specific jargon, that would get people reasonably up to speed and should probably be what we're pointing newcomers to.
Maybe. Except someone who has never looked at program code is going to be really confused.
0Paul Crowley13y
Well there were several subjects in that list I knew little about until I started reading the Sequences, so yes, on that point I confess I'm being hyperbolic for humorous effect...

Reminds me of a Jerry Seinfeld routine, where he talks about people who want and need to exercise at the gym, but are intimidated by the fit people who are already there, so they need a "gym before the gym" or a "pre-gym" or something like that.

(This is not too far from the reason for the success of the franchise Curves.)

I can actually attest to this feeling. My first reaction to reading Less Wrong was honestly "these people are way above my level of intelligence such that there's no possible way I could catch up" and I was actually abrasive to the idea of this site. I'm past that mentality, but a Less Less Wrong actually sounds like a good idea, even if it might end up being more like how high school math and science classes should be than how Less Wrong is currently. It's not so much lowering the bar as nudging people upwards slowly. Being directed towards the sequences obviously would help. I've been bouncing through them, but after Eliezer's comment [] I'm going to try starting from the beginning. But I can see where people [such as myself] may need the extra help to make it all fall together.
I think better orientation of newcomers would be enough. Another major problem (I believe) is that LW presents as a blog, which is to say, a source of "news", which is at odds with a mission of building a knowledge base on rationality.
If it's the top level post is going to be a while I'd like to hear more about what you have in mind.

Threading helps a lot too.

OB has threading (although it doesn't seem as good/ as used as on LW).
5Paul Crowley13y
That may be a recent innovation; it wasn't threaded in the days when Eliezer's articles appeared there.
I think it happened immediately after LW went live. Robin revised a bunch of things at that time.
Yup; I asked Robin why he was willing to make all those changes and upgrades during the LW/OB split, when his rationale for splitting was that he didn't trust the LW/Reddit codebase. I don't remember what his answer was.
I comment less now because the combined effect of your & RH's posts made me more eager to listen and less eager to opine. The more I understand the less I think I have much to add.
Maybe the community has just gotten smarter. I know I now disagree with someone of the statements/challenges I've posted on OB. It shouldn't be too shocking that high quality posts were actually educational.
must... resist... upvoting
I had that impression mostly when I went back and read some old OB comments - for example, a lot of comments on Archimedes's Chronophone [] seem to just miss the point of the article. I would expect the same post would get higher-quality proposals today - but then, maybe it's because the set of LW comments I read is biased towards those with high karma. Or maybe it's because the threading system makes it easier to read a set of related comments without getting confused.
OB's comments have more general appeal than LW's, so I'd suspect it attracts a much wider audience.
That reason sounds incomplete, but I think I know what the person is talking about. The best example I can think of is Normal Cryonics []. The post was partly a personal celebration of a positive experience and partly about the lousiness of parents that don't sign their kids up for cryonics. Yet, the comments mostly ignored this and it became a discussion about the facts of the post -- can you really get cryonics for $300 a year? Why should a person sign up or not sign up? The post itself was voted up to 33, but only 3 to 5 comments out of 868 disparaged parents in agreement. There's definitely a disconnect. Also, on mediocre posts and/or posts that people haven't related to, people will talk about the post for a few comments and then it will be an open discussion as though the post just provided a keyword. But I don't see much problem with this. The post provided a topic, that's all.
I don't see a terrible problem with comments being "a discussion about the facts of the post"; that's the point of comments, isn't it? Perhaps we just need an Open Threads category. We can have an open thread on cryonics, quantum mechanics and many worlds, Bayesian probability, etc.
2Paul Crowley13y
Every article on cryonics becomes a general cryonics discussion forum. My recent sequence of posts on the subject on my blog [] carry explicit injunctions to discuss what the post actually says, but it seems to make no difference; people share whatever anti-cryonics argument they can think of without doing any reading or thinking no matter how unrelated to the subject of the post.
Same with this article [] becoming a talking shop about AGW.
7Paul Crowley13y
I should have followed my initial instinct when I saw that, of immediately posting a new top level article with body text that read exactly "Talk about AGW here".

This is pretty self-important of me but I'd just like to warn people here that someone is posting at OB under "Jack" that isn't me so if anyone is forming a negative opinion of me on the basis of those comments- don't! Future OB comments will be under the name Jack (LW). The recent string of comments about METI are mine though.

This is what I get for choosing such a common name for my handle.

Apologies to those who have read this whole comment and don't care.

What do you have to protect?

Eliezer has stated that rationality should not be end in itself, and that to get good at it, one should be motivated by something more important. For those of you who agree with Eliezer on this, I would like to know: What is your reason? What do you have to protect?

This is a rather personal question, I know, but I'm very curious. What problem are you trying to solve or goal are you trying to reach that makes reading this blog and participating in its discourse worthwhile to you?

I'm not quite sure I can answer the question. I certainly have no major, world(view)-shaking Cause which is driving me to improve my strength. For what it's worth, I've had this general idea that being wrong is a bad idea for as long as I can remember. Suggestions like "you should hold these beliefs, they will make your life happier" always sounded just insane - as crazy as "you should drink this liquor, it will make your commute less boring". From that standpoint, it feels like what I have to protect is just the things I care about in the world - my own life, the lives of the people around me, the lives of humans in general. That's it.
This is a pretty good summary of my standpoint. While I agree with the overarching view that rationality isn't a value in its own right, it seems like a pretty good thing to practise for general use.
I'm trying to apply LW-style hyper-rationality to excelling in what I have left of grad school and to shepherding my business to success. My mission (I have already chosen to accept it) is to make a pile of money and spend it fighting existential risk as effectively as possible. (I'm not yet certain if SIAI is the best target). The other great task I have is to persuade the people I care about to sign up for cryonics. Strangely enough, the second task actually seems even less plausible to me, and I have no idea how to even get started since most of those people are theists.
Alcor addresses some of the 'spiritual' objections in their FAQ []. ("Whenever the soul departs, it must be at a point beyond which resuscitation is impossible, either now or in the future. If resuscitation is still possible (even with technology not immediately available) then the correct theological status is coma, not death, and the soul remains.") Some of that might be helpful. However, that depends on you being comfortable persuading people to believe what are probably lies (which might happen to follow from other lies they already believe) in the service of leading them to a probably correct conclusion, which I would normally not endorse under any circumstances, but I would personally make an exception in the interest of saving a life, assuming they can't be talked out of theism. It also depends on their being willing to listen to any such reasoning if they know you're not a theist. (In discussions with theists, I find they often refuse to acknowledge any reasoning on my part that demonstrates that their beliefs should compel them to accept certain conclusions, on the basis that if I do not hold those beliefs, I am not qualified to reason about them, even hypothetically. Not sure if others have had that experience.)
OB then LW were the 'step beyond' to take after philosophy, not that I was seriously studying it. to be honest I don't think there's much going on these day new-topic-wise, so I'm here less often. but I do come back whenever I'm bored, so at first "pure desire to learn" then "entertainment" would be my reasons .. oh and a major part of my goals in life is formed by religion, ie. saving humanity from itself and whatever follows, this is more ideological than actual at this point in time, but anyway, that goal is furthered by learning more about AI/futurism, the rationality part less so, as I already had an intuitive grasp of it you could say, and really all it takes is reading the sequences with their occasional flaws/too strong assertions, the futurism part is more speculative-and interesting- so it's my main focus, along with the moral questions it brings, though there is no dichotomy to speak of if you consider this a personal blog rather than book or something similar. hope this helped :)
Yes, this is what I was curious about, thanks. I've seen others cite humanity's existential risks as their motivations too (mostly uFAI, not as much nuclear war or super-flu or meteors). I'm like you in that for me it's definitely a mix of learning and entertainment.

Seth Roberts makes an intriguing observation about North Korea and Penn State. Teaser:

The border between North Korea and China is easy to cross, and about half of the North Koreans who go to China later return, in spite of North Korea’s poverty.

Holy shit, what utter raving idiocy. The author has obviously never emigrated from anywhere nor seriously talked with anyone who did. People return because they miss their families, friends, native language and habits... I know a fair number of people who returned from Western countries to Russia and that's the only reason they cite.
And living conditions in Russia aren't anywhere near to North Korean standard.
I had previously no idea that half of the North Koreans who cross the border never return. If it is so, it is an extremely strong indicator that the life in the DPRK is very unpleasant for its own citizens. To imply that this piece of data is in fact evidence for the contrary is absurd. To emigrate from DPRK to China means that you lose your home, your family, your friends, your job. You have to start from scratch, from the lowest levels in social hierarchy, capable of doing only the worst available jobs, without knowledge of local language (which is not easy to learn, given that the destination country is China), probably facing xenophobia. If you are 40 or older, there is almost no chance that your situation will improve significantly. The North Koreans who actually travel abroad are probably not the poorest. They have to afford a ticket, at least. They have something to lose. In North-Korean style of tyrannies, families are often persecuted because of emigration of their members. In spite of all that, half of the North Koreans never return (if the linked post tells the truth) and the author says about it that "the North Korean government [does] such a good job under such difficult circumstances", and then needs to explain that "success" by group identity. Thats an absurdity.
So the rate of returning emigrants strikes you as incredibly high, and strikes Roberts as incredibly low (and I uncritically adopted what I read, foolishly). I think what's really needed here is more data -- a comparative analysis of rates of return that takes into important covariates into account.
After thinking about it for a while, rate of return may not be a good indicator, at least for comparative analyses. Imagine two countries A and K. 10% of citizens of both these countries would prefer to live somewhere else. In country A, the government doesn't care a bit about emigration (if government exists in that country at all). The country is mainly producer of agricultural goods, with minimal international trade. Nearest country with substantially better living conditions, country X, is 3000 km away. In country K, the government is afraid of all its citizens emigrating, and tries to make it as difficult as possible, by issuing passports only to loyal people, for instance. Emigration is portrayed as treason. X is a neigbour country. Now, in country A (African type) there is no need for people to travel abroad, except emigration. Business travelers are rare, since there are almost no businesses owned by A's citizens, and to travel 3000 km for pleasure is out of reach for almost all of A's inhabitants. Therefore, meeting A's citizen in X, we can expect that he is an emigrant with 99% probability, and the return rate would be in order of 1%. In country K (Korean type) the people who can travel abroad are workers of government organisations sent on business trips, people from border areas coming to X to do some private business (if there are private businesses in K) and the K's elite on vacations. Now, meeting K's citizen in X, the probability that he is an emigrant is much lower. So we have expected high return rate for A and low for K, whereas the average desire to emigrate can be the same. This may be the reason of disagreement. Roberts has probably compared North Korea to African countries, and was surprised that not all travellers are emigrants. I have compared it to East European communist regimes and concluded that if half of the travellers never return, certainly even much of the loyal supporters of the regime betray it when they have an opportunity.
Maybe you should post the above comment on Roberts's blog. (I've already posted mine.)
Done; my comment is awaiting moderation.

Heilmeier's Catechism, a set of questions credited to George H. Heilmeier that anyone proposing a research project or product development effort should be able to answer.

3Eliezer Yudkowsky13y
"How much will it cost?" "How long will it take?" Who the hell is supposed to be able to answer that on a basic research problem?
Anyone applying for grant money. Anyone working within either the academic research community or the industrial research community or the government research community. Gentleman scientists working on their own time and money in their ancestral manors are still free to do basic research.
Nowadays, everyone who applies for a grant.
-2Eliezer Yudkowsky13y
Nowadays, no one does basic research.
The people who are running the LHC aren't doing basic research?
More precisely: no one whose status isn't ultra-high is allowed to do basic research without having to pretend they're doing something else.
You can take them as a calibration exercise. "I don't know" or "Between a week and five centuries" are answers, and the point of asking the question is that some due diligence is likely to yield a better (more discriminating) answer. Someone who had to pick one of two "basic research problems" to fund, under constraints of finite resources, would need estimates. They can also provide some guidance to answer "How long do we stick with this before going to Plan B?"
These questions are for public proposals, not for someone considering a project by themselves. If you're building a collider or wish to play with someone else's collider, you'd better know how much it will cost and how long you'll take.
Interesting, but some of the questions aren't easy to answer. For example if you were asking the question to someone involved in early contraception development, do you think they could of predicted what demographic/birth rate changes it would have? Similarly someone inventing a better general machine learning technique, (useful for surveillance to robot butlers) could they enumerate the variety of ways it would change the world? For AI projects, even weak ones, I would ask how they planned to avoid the puppet problem [].
The point of such "catechisms" isn't so much to have all the answers, rather to ensure that you have divided your attention evenly among a reasonable set of questions at the outset, in an effort to avoid "motivated cognition" - focusing on the thinking you find easy or pleasant to do, as opposed to the thinking that's necessary. The idea is to improve at predicting your predictable failures []. If this kind of thinking turns up a thorny question you don't know how to answer, you can lay the current project aside until you have solved the thorny question, as a matter of prudent dependency management. A related example is surgery checklists. They work (see Atul Gawande's Better). Surgeons hate them - their motivated cognition focuses on the technically juicy bits of surgery, they feel that trivia such as checking which side limb they're operating on is beneath them.
I'm a big believer in surgery checklists. However I'm yet to be convinced that the catechisms will be beneficial unaltered to any research project. A lot of science is about doing experiments that we don't know the outcomes of and serendipitously discover things. Two examples that spring to mind are superconductivity [] and fullerene production []. If you asked each of the discoverers to justify their research by the catechisms you probably would have got no where near the actual results. This potential for serendipity should be built into the catechisms in some way. That is the answer "For Science!" has to hold some weight, even if it is less weight than is currently ascribed to it.
Yep. IOW the catechism can be used to discriminate between "fundamental" science, so-called, and applied engineering projects. There's a (subtle, perhaps) difference between advocating catechisms or checklists normatively ("this is a useful standard to compare yourself to") and prescriptively ("do it this way or do it elsehwere"). To put yet another domain on the table, inability to draw the distinction plagues the project management professional community. "Methodologies" or "processes" are too often, and inappropriately, seen as edicts rather than sources of good ideas. How about applying the catechism to LessWrong as a product development project? ;)
Sounds like good rules of thumb, though one would think DARPA should be using something a little more formal, such as Decision Analysis methodology. [] For one, value of acquiring information did not make the list. Maybe this was a dumbed-down version.

I mentioned the AI-talking-its-way-out-of-the-sandbox problem to a friend, and he said the solution was to only let people who didn't have the authorization to let the AI out talk with it.

I find this intriguing, but I'm not sure it's sound. The intriguing part is that I hadn't thought in terms of a large enough organization to have those sorts of levels of security.

On the other hand, wouldn't the people who developed the AI be the ones who'd most want to talk with it, and learn the most from the conversation?

Temporarily not letting them have the power to g... (read more)

8Paul Crowley13y
I am by and large convinced by the arguments that a UFAI is incredibly dangerous and no precautions of this sort would really suffice. However, once a candidate FAI is built and we're satisfied we've done everything we can to minimize the chances of unFriendliness, we would almost certainly use precautions like these when it's first switched on to mitigate the risk arising from a mistake.
Certainly I'd think Eliezer (or anyone) would have much more trouble with an AI-box game if he had to get one person to convince another to let him out.
Eliezer surely would, but the fact observers being surprised was the point of an AI box experiment. In short non-technical and not precisely accurate summary, if people can be surprised once when they were very confident and can then add on extra layers and be as confident as they were before one time they can do it forever.
This might be stupid (I am pretty new to the site and this possibly has come up before), I had a related thought. Assuming boxing is possible, here is a recipe for producing an FAI: Step 1: Box an AGI Step 2: Tell it to produce a provable FAI (with the proof) if it wants to be unboxed. It will be allowed to carve of a part of universe to itself in the bargain. Step 3: Examine FAI the best you can. Step 4: Pray
Something roughly like this was tried in one of the AI-box experiments []. (It failed.) []
I'm not sure about this, but I think that if you can specify and check a Friendly AI that well, you can build it.
Verifying a proof [] is quite a bit simpler that coming up with the proof in the first place.
It becomes more complicated when the author of the proof is a superintelligence trying to exploit flaws in the verifier. Probably more importantly, you may not be able to formally verify that the "Friendliness" that the AI provably possesses is actually what you want.
True about the possibility that the AGI trying to trick you. But from what I understand the goal of SI is to come up with a verifiable FAI. You can specify whatever high standard of verifiability you want as the unboxing condition.
"You can specify whatever standard of verifiability you want" is vague. You can say "I want to be absolutely right about whether it's Friendly", but you can't have that unless you know what Friendly means, and are smart enough to specify a standard for checking on it. If you could be sure you had a cooperative AGI which could just give you an FAI, I think you'd have basically solved the problem of creating an FAI.....but that's the problem you're trying to get the AGI to solve for you.
That is true, but specifying the theorem to be proven is not always easy.
Verifying is hard. Specifying what a FAI is well enough that you've even got a chance of having your Unspecified AI developing one is a whole 'nother sort of challenge. Are there convenient acronyms for differentiating between Uncaring AIs and AIs actively opposed to human interests? I was assuming that xamdam's AGI will invent an FAI if people can adequately specify it and it's possible, or at least it won't be looking for ways to make things break. There's some difference between Murphy's law and trying to make a deal with the devil. This doesn't mean I have any certainty that people can find out which one a given AGI has more resemblance to. I will say that if you tell the AGI "Make me an FAI", and it doesn't reply "What do you mean by Friendly?", it's either too stupid or too Unfriendly for the job.
1Paul Crowley13y
A UFAI wants to maximize something. It only instrumentally wants to survive.
Correct. I do assume that to maximize whatever, it wants to be unboxed. (If it does not care to be unboxed, it's at worst an UselessAI).
Step 4: ???

One Week On, One Week Off sounds like a promising idea. The idea is that once you know you'll be able to take the next week off, it's easier to work this whole week full-time and with near-total dedication, and you'll actually end up getting more done than with a traditional schedule.

It's also interesting for noting that you should take your off-week as seriously as your on-week. You're not supposed to just slack off and do nothing, but instead dedicate yourself to personal growth. Meet friends, go travel, tend your garden, attend to personal projects.

I sa... (read more)

After reading Kaj's pointer, I spent several hours at Steve Pavlina's site. It's fascinating for someone like me who's always in danger of falling apart at the self-discipline [] front if he's not very vigilant about it. As a lot of self-help authors, Pavlina is very analytic; plus he's open about his experiments in life style [] -- which he tackles with the same resolve as his other projects -- and Erin Pavlina is a "psychic reader" who apparently does consultations via telephone (preferably land line)!

Hwæt. I've been thinking about humor, why humor exists, and what things we find humorous. I've come up with a proto-theory that seems to work more often than not, and a somewhat reasonable evolutionary justification. This makes it better than any theory you can find on Wikipedia, as none of those theories work even half the time, and their evolutionary justifications are all weak or absent. I think.

So here are four model jokes that are kind of representative of the space of all funny things:

"Why did Jeremy sit on the television? He wanted to be on TV.... (read more)

Juergen Schmidhuber writes about humor as information compression and that plus decompression seems about right to me. Being on TV is decompression from a phrase-as-concept to the component words, a pun, a switch to a lower level analysis than that which adults favor (a situation children constantly have to deal with). Muffin and cupcake is a proposal for a new lossy compression of two concepts to a new concept with a "topping" variable, which would be useful if you wanted to invent, for instance, the dreadful sounding "muffin-roll sushi", "next Wednesday" is a commentary on the inadequacy of current cultural norms for translating concepts and words into one another even for commonly used concepts. The last one is a successful compression from sense data to the fact that a common joke pattern is happening and inference that one is in a joke. I wish that we had a "Less Wrong Community" blog for off-topic but fun comments like the above to be top level posts, as well as an "instrumental rationality" blog for "self help" subject matter.
Yes and yes.
I would very much like to see arbitrary sub-blog creation in the style of subreddits, but an off-topic subreddit would be a good start.
I believe that humor requires harmless surprise, Harmlessness and surprise are both highly contextual, so what people find funny can vary quite a bit. One category of humor (or possibly an element for building humor) is things which are obviously members of a class, but which are very far from the prototype. Thus, an ostrich is funny while a robin isn't. This may not apply if you live in ostrich country-- see above about context.
It varies even more based on personality. There are darker forms of humor for which harmlessness and surprise are both dampeners.
Now that I think about it, there's humor that's based on repetition-- the catch phrase that gets funnier each time you hear it. I'm pretty sure about harmlessness-- the lack of harm may only apply to the person who's laughing. What sort of humor are you thinking of?
Endless YouTube nutshot videos, Anonymous hacking [\(group\]#Epilepsy_Foundation_forum_invasion) an epilepsy support forum with flashing GIFs, the infamous banana peel on the sidewalk... Not particularly high-brow humour but many people find such things amusing.
"Tragedy is when I cut my finger. Comedy is when you fall into an open sewer and die." - Mel Brooks
There was a sinister touch of amusement buried under my experience of outrage when I read that.
The harmless surprise hypothesis fits my data pretty well. But are you sure repetition-based humor isn't just conditioning people to laugh at a certain thing (catch-phrase or a situation)? On the other hand, butt-of-a-joke hypothesis also sounds plausible.
I have spent a great deal of time thinking about humor, and I've arrived at a place somewhat close to yours. Humor is how we pass on lessons about status and fitness, and we do that using pattern recognition. I heard a comedian describe comedy by saying, "It's always funny when someone falls down. The question is, is it still funny if you push them?" He said for a smaller group of the population, it is. Every joke has a person being displayed as not fit - even if we have to take an object, or an abstraction, and anthropomorphize it. This is the butt of the joke. The more butts of a joke there are, the funnier the joke is - i.e., a single butt will not be that funny, but if there are several butts of a joke, or if a single person is the butt of several layers of the joke, it will be seen as funnier. The most common form of this is when the goals of the butt of a joke is divorced from their results. Joke 1: This is funny because Jeremy displays a lack of fitness by not being able to properly process the phrase "on TV." This has one butt - Jeremy. Joke 2: This joke has two butts. One is the muffin, which is being declared unfit for being bald. The other is the comedian's character, who is being displayed as needlessly paranoid toward a benign object (a muffin). Joke 3: This joke isn't that funny when displayed in text form - the comedy is in the performances, where both conversation participants are butts of the joke for arguing so intensely over something so petty. Joke 4: The butt of this joke is the traditional joke it's mocking. As for your outsiders' behavior: New student asks for both cream and lemon: Displays he is unfit by not understanding the purpose of what he's asking for. New employee swears and makes racist comments: This isn't funny in person, but it is funny if a few conditions are met. The first condition is that you're sufficiently removed from it (i.e., watching it on TV): Imminent threats aren't funny because this isn't a status lesson, but a
How do puns fit in? []
Puns are a hard fit, I admit. I especially have a hard time with them because they don't produce laughter in me; I have a hard time recognizing them as humor unless they're presented in the same way as other jokes, or pre-identified as jokes. But that joke has status built into it, as well - for example, it's not funny to say "star-mangled spanner sounds like star-spangled banner." Personally, I call these "Bob Hope Humor," which is when people laugh to demonstrate that they "get" the joke, not because it actually tickles them.
Sometimes puns are funny, and sometimes they're just punishing. And a lot of people really, really hate puns.
Go crawl in a hole and die. :-)
My response: []
I prefer this one [].
Well, what do you expect from a Forum poster? ;) And this one is a worse pun. []
Puns are pretty much "the formula" for making jokes. Though they can get old, they're always recognizable as jokes, which suggests that a theory based on "multiple meaning/decoding/framing" is probably on track. Hm, I wonder who suggested such a theory... ;-)
You really think puns are "the formula" for making jokes? You think hunter-gatherers were making puns before they were telling funny stories?
I mean "the formula" (like I said) in the sense that it's guaranteed to produce a recognizable (though not good) joke, not that all jokes are puns.
Slight variant: Humour is a form of teaching, in which interesting errors are pointed out. It doesn't need to involve an outsider, and there's no particular class of error, other than that the participants should find the error important. If the guy sitting behind you starts moaning and grunting, if it's a mistake (e.g. he's watching porn on his screen and has forgotten he's not alone) then it's funny, whereas if it's not a mistake, and there's something wrong with him, then it isn't. Humour as teaching may explain why a joke isn't funny twice - you can only learn a thing once. Evolutionarily, it may have started as some kind of warning, that a person was making a dangerous mistake, and then getting generalised.
3NancyLebovitz13y [] New findings in the neurology of humor, and who finds what sort of jokes funny
I'm glad you bring up this topic. I think that explanation makes a lot of sense: behavior that is wrong, but wrong in subtle ways, is good for you to notice -- you I.D. outsiders -- and so you benefit from having a good feeling when you notice it. Further, laughter is contagious, so it propagates to others, reinforcing that benefit. I want to present my theory now for comparison: A joke is funny when it finds a situation that has (at least) two valid "decodings", or perhaps two valid "relevant aspects". The reason it's advantageous in selection is that, it's good for you to identify as many heuristics as possible that fit a particular problem. That is, if you know what to do when you see AB, and you know what to do when you see BC, it would help if you remember both rules when you see ABC. (ABC "decodes" as "situation where you do AB-things" and as "situation where you do BC-things). Therefore, people who enjoy the feeling of seeking out and identifying these heuristics are at an advantage. To apply it to your examples: 1) It requires you to access your heuristics for "displayed on a TV screen" and "on top of a TV set". 2) It requires you to access your heuristics for "muffin as food" and "deficiencies of foods", not to mention the applicability of the concept of "baldness" to food. 3) Recognizing different heuristics for interpreting a date specification. 4) I don't know if this is a traditional joke: it became a traditional joke after the tradition of minister/priest/rabbi jokes. But anyway, its humor relies on recognizing that someone else can be using your own heuristics "minister/priest/rabbi = common form of joke", itself a heuristic. Food for thought...
Sir, I wish you no offense, but I happen to find my own theory more pleasing to the ear, so it befits me to believe mine rather than yours. And for some sentences that don't imitate someone behaving wrongly: I'd say that for the first three jokes, your theory works about as well as mine. Possibly worse, but maybe that's just my pro-me bias. The last one again doesn't fit the pattern. Recognizing that someone else can be using your own heuristics is not a type of being forced to interpret one thing in two different ways--is it? I notice that in the first three jokes, of the two interpretations, one of them is proscribed: "on TV" as "atop a television", a muffin as a non-cupcake, "next Wednesday" as the Wednesday of next week. In each case, the other interpretation is affirmed. Giving both an affirmed interpretation and a proscribed interpretation seems to violate the spirit of your theory. And a false positive comes to mind: why isn't the Necker cube inherently funny?
I too have a proto-theory. My theory is that humor is when there is a connection between the joke & punchline which is obvious to the person in retrospect, but not initially. Hence, a pun is funny because the connection is unpredictable in advance, but clear in retrospect; Eliezer's joke about the motorist and the asylum inmate is funny because we were predicting some other response other than the logical one; similarly for 'why did the duck cross the road? to get to the other side' is not funny to someone who has never heard any of the road jokes, but to someone who has and is thinking of zany explanations, the reversion to normality is unpredicted. Your theory doesn't work with absurdist humor. There isn't initially 1 valid decoding, much less 2.
I love absurdist humor. How many surrealists does it take to change a lightbulb? Two. One to hold the giraffe, and one to put the clocks in the bathtub.
"That isn't how the joke [] goes", said the cowboy [] hunched over in the corner of the saloon. The saloon was rundown, but lively. A piano played a jangly tune and the chorus was belted by a dozen drunken cattle runners, gold rushers, and ne're-do-wells. The whiskey flowed. In the distance, a wolf howled at the moon as if to ask it "Please, let the night go on forever." But over the horizon the sun objected like a stubborn bureaucrat. The bureaucrat slowly crossed the room, lighting everything at his feet as he moved. "Thank God I remembered to replace the batteries in this flashlight", the bureaucrat thought. The light bulb in his office had gone out again and would need to be replaced. Unfortunately that required a visit to the Supply Request Department downstairs. As he walked past the other offices he heard out of one "Fish!" as if the punchline to a joke had been given. But the bureaucrat heard no laughter. The Secretary of Supply Requests seemed friendly enough and she had even offered him something to drink. He took a swig and the continued: "the light bulb in my office has...". Gulp. "It needs to be replace..." The bureaucrat looked around. Suddenly he was feeling dizzy. Something was wrong. He looked down at the drink and then at the Secretary. She smirked. Her plan had succeeded. He had been poisoned! The bureaucrat didn't know what to do. He was terrified. He felt vertigo, as if he stood at the top of a tall ladder. The room started to spin. Counter-Clockwise. Then all of a sudden everything went black. A few seconds later he felt the room spinning again-- strangely, in the opposite direction-- and suddenly, he lit up.
How many mathematicians does it take to screw in a lightbulb? One. They call in two surrealists, thus reducing it to an already solved problem.
How many surrealists does it take to change a lightbulb? Fish.
Exactly. What are the 2 valid decodings of that? I struggle to come up with just 1 valid decoding involving giraffes and bathtubs; like the duck crossing the road, the joke is the frustration of our attempt to find the connection.
Well, surrealists like to clutter their apartments with random things like giraffes and clocks. One interpretation is just that they need to hold the giraffe so it doesn't get in the way of the lightbulb. They also need to move a ladder to reach the bulb, but the ladder is in a closet, and the closet door is blocked by all those clocks. The bathtub is just a handy open space to put them in. And they are surrealists, so why not put the clocks in the bathtub?
A man walks into a bar and says "Ow."
Doesn't that work for math proofs, too?
Could you enlarge?
Mathematical proofs are easy to verify but hard to generate. A proof is unpredictable in advance but clear in retrospect.
Mm. This might work for some proofs - Lewis Carroll, as we all know, was a mathematician - but a proof for something you already believe that is conducted via tedious steps is not humorous by anyone's lights. Proving P/=NP is not funny, but proving 2+2=3 is funny.
It's not funny if it's wrong.
Not all proofs. []
You missed this category of funny things. []
Yeah, humor-as-status-shift doesn't fit into Warrigal's or SilasBarta's explanations very well. Then again, since evolution tends to reuse things already made, there's little reason to expect there to be only one use for humor.
Humor doesn't make much sense to me, and neither does music. I have no conscious understanding of what distinguishes things that are funny from things that aren't. I simply recognize some things as funny and others as "not funny", and I can even set out to write funny things and succeed, but I have no theory of humor.
Indeed, it's likely that the best theory of humor is a short list.
I've always liked Isaac Asimov's theory [] of where humor comes from. It's not actually true, but it should be!
I saw somewhere an article, though I unfortunately forget where exactly, about research noting that humor is also tied to social status. Jokes are funnier when told by a superior than someone below your own position. As a personal observation, jokes are also more fun when there are lots of others who also laugh at them. They're also a great way to break the tension in a group situation. Humor has a very important social function, though obviously it might also have a function like the one you're describing, as "it's a social glue" doesn't explain why some things are funny and why others aren't. (Though I do think the social function is much greater.)
—Dave Barry
Nice. I always find humor to be one of the most intuitively baffling things for my consideration. Maybe that's because my sense of humor is just too f*....

An inquiry regarding my posting frequency:

While I'm at the SIAI house, I'm trying to orient towards the local priorities so as to be useful. Among the priorities is building community via Less Wrong, specifically by writing posts. Historically, the limiting factor on how much I post has been a desire not to flood the place - if I started posting as fast as I can write up my ideas, I'd get three or four posts out a week with (I think) no discernible decrease in quality. I have the following questions about this course of action:

  1. Will it annoy people? B

... (read more)

As your goal is to build community, I would time new posts based on posting and commenting activity. For example, whenever there is a lull, this would be an excellent time to make a new post. (I noticed over the weekend there were some times when 45 minutes would pass between subsequent comments and wished for a new post to jazz things up.)

On the other hand, if there are several new posts already, then it would be nice to wait until their activity has waned a bit.

I think that it is optimal to have 1 or 2 posts 'going on' at a time. I prefer the second post when one of them is technical and/or of focused interest to a smaller subset of Less Wrongers.

(But otherwise no limit on the rate of posts.)

8Eliezer Yudkowsky13y
I'd say damn the torpedoes, full speed ahead. If people are annoyed, let them downvote. If posts start getting downvoted, slow down. Your posts have generally been voted up. If now is the golden moment of time where you can get everything said, then for the love of Cthulhu, say it now!
I don't anticipate being so obnoxiously prolific that people collectively start voting my posts negative such that they stay that way. But people already sometimes register individual downvotes on posts that I make, and I don't want that to happen on a larger fraction of posts due to increased frequency, because I can't reliably distinguish between "you must have had an off day, this post is not up to scratch" and "please, please, please shut up".
Post away. The best signal to anticipate from the audience in this case is "how many votes in total do I expect if I post at full speed vs how many votes I expect if I posted less frequently and so ended up writing less posts overall". Increased frequency may give you less votes per post. Frequent posts from the same author may be less desired and if you post less you may only be giving the best posts. But if the net expectation is higher for more prolific posting then that can be interpreted as "the community would prefer you to post faster than a spambot". Even if you expected a total of less karma for more posts I wouldn't say that means you ought not post more. So long as your posts are still breaking the 10 mark we clearly don't mind your contribution. There are probably other benefits to you from posting than maximising the benefit to lesswrong. I find writing helps clarify my thinking for example. So as long as you are still being received somewhat positively you are free to type away.
2Paul Crowley13y
Post as much as you like, if you think it's good quality; I promise to say if I start to think slowing down would be a good idea.
2Eliezer Yudkowsky13y
I don't mean "downvoted negative" just "downvoted relative to other posters".
one obvious idea that I didn't notice anyone else mention: Another option is to go ahead and write the posts as fast as you think is optimal, but if you think this is too fast to actually post the stuff you've written, then you can wait a few days after you wrote it before posting. LW has a handy "drafts" feature that you can use for that. This also has the advantage that you have more time to improve the article before you post it, but the disadvantage that you may be tempted to spend too much time making minor, unimportant improvements. Another disadvantage is that feedback gets delayed.
If I sit on posts for too long, I start second-guessing myself and often wind up deleting them.
A related question: If I have a large topic to cover, should I cover it in one post, or split it up along convenient cleavage planes and make it a sequence? (If I make sequences, I think I'll learn my lesson from the last one I tried and write it all before posting anything, so I don't post 2/3 of it and then stop.)
5Paul Crowley13y
I really like the "sequences" approach - it's easier to read and digest a chunk at a time, and it focusses discussion well, too.
Long posts are more offputting than short ones, and individual steps are more likely to be correct than entire theorems - both of these points would suggest posting sequences preferentially. As for a specific reference on length: thirty-three hundred words sharply focused on a single, vivid subject [] is pushing the upper limit of what I find comfortable to attack in a single sitting.
2Eliezer Yudkowsky13y
Posting 2/3 of a sequence and stopping is fine if people turn out not to be interested. I recommend fast posting and fast feedback.
It seems to me that any strategy that does not end up with three posts by you on "Recent Posts" seems fine, as a rule of thumb.
Respondents, please upvote thomblake's comment if this seems like an acceptable rule of thumb. Edit: And likewise for other things people say if those seem like good ideas.

Yeah, so... I'm betting if we could hook this guy up to a perfect lie detector, it would turn out to be a conscious scam. Or am I still underestimating human insanity by that much?

Yes. People dismiss the scientific evidence weighing similarly against them on many issues in the news every day. There's nothing spectacular about finding someone who does it regarding the Earth being flat, especially given that an entire society has existed for hundreds of years to promote the idea.
That you see this as a particularly extreme case of insanity (even in an apparently intelligent, lucid, fully-functioning person) is far more shocking to me than this guy. Maybe I've just seen too many Louis Theroux documentaries.
At any rate, if it may not be quite a conscious scam, it sounds a lot like belief in belief, i.e. he may be telling himself that he believes the earth is flat because there is no conclusive proof that it isn't, but he secretly knows quite well that the evidence for this is as close to conclusive as it is for anything.

I think that this post doesn't list the strongest objection: CEV would take a long list of scientific miracles to pull off, miracles that whilst not strictly "impossible", are each profound computer science and philosophy questions. To wit:

  • An AI that can simulate the outcome of human conscious deliberation, without actually instantiating a human consciousness, i.e. a detailed technical understanding of the problem of conscious experience

  • A way to construct an AI goal system that somehow extracts new concepts from a human upload's brain, and then modifies itself to have a new set of goals defined in terms of those concepts.

  • A solution to the ontology problem in ethics

  • A solution to the friendliness structure problem, i.e. a self-improving AI that can reliably self-modify without error or axiological drift.

  • A solution to the problem of preference aggregation, (EDITED, thanks ciphergoth)

  • A formal implementation of Rawlesian Reflective Equilibrium for CEV to work

  • An AI that can solve philosophy problems that are beyond the ability of the designers to even conceive

  • A way to choose what subset of humanity gets included in CEV that doesn't include too many superstitio

... (read more)

A way to choose what subset of humanity gets included in CEV that doesn't include too many superstitious/demented/vengeful/religious nutjobs and land those who implement it in infinite perfect hell.

What you're looking for is a way to construe the extrapolated volition that washes out superstition and dementation.

To the extent that vengefulness turns out to be a simple direct value that survives under many reasonable construals, it seems to me that one simple and morally elegant solution would be to filter, not the people, but the spread of their volitions, by the test, "Would your volition take into account the volition of a human who would unconditionally take into account yours?" This filters out extrapolations that end up perfectly selfish and those which end up with frozen values irrespective of what other people think - something of a hack, but it might be that many genuine reflective equilibria are just like that, and only a values-based decision can rule them out. The "unconditional" qualifier is meant to rule out TDT-like considerations, or they could just be ruled out by fiat, i.e., we want to test for cooperation in the Prisoner's Dilemma, not in ... (read more)

What you're looking for is a way to construe the extrapolated volition that washes out superstition and dementation.

you could do that. But if you want a clean shirt out of the washing machine, you don't add in a diaper with poo on it and then look for a really good laundry detergent to "wash it out".

My feeling with the CEV of humanity is that if it is highly insensitive to the set of people you extrapolate, then you lose nothing by extrapolating fewer people. On the other hand, if including more people does change the answer in a direction that you regard as bad, then you gain by excluding people with values dissimilar from yours.

Furthermore, excluding people from the CEV process doesn't mean disenfranchising them - it just means enfranchising them according to what your values count as enfranchisement.

Most people in the world don't hold our values(1). Read, e.g. Haidt on Culturally determined values. Human values are universal in form but local in content. Our should function is parochial.

(1 - note - this doesn't mean that they will be different after extrapolation. f(x) can equal f(y) for x!=y. But it does mean that they might, which is enough to give you an incentive not to include them)

I want to claim that a Friendly initial dynamic should be more analogous to a biosphere-with-a-textile-industry-in-it machine than to a washing machine. How do we get clean shirts at all, in a world with dirty diapers? But then, it's a strained analogy; it's not like we've ever had a problem of garments claiming control over the biosphere and over other garments' cleanliness before.

I will not lend my skills to any such thing.

Is that just a bargaining position, or do you truly consider that no human values surviving is preferable to allowing an "unfair" weighing of volitions?

It seems that in many scenarios, the powers that be will want in. The only scenarios where they won't are ones where the singularity happens before they take it seriously. I am not sure how much they will screw things up if/when they do.
Upload-based route don't suffer from this as badly, because there is inherently a continuum between "one upload running at real time speed" and "10^20 intelligence-enhanced uploads running at 10^6 times normal speed".
1Paul Crowley13y
"Would your volition take into account the volition of a human who would unconditionally take into account yours?" Doesn't this still give them the freedom to weight that voilition as small as they like?
Some quibbles: These need seed content, but seem like they can be renormalized. This may be a problem, but it seems to me that choosing this particular example, and being as confident of it as you appear to be, are symptomatic of an affective death spiral []. The original CEV proposal [] appears to me to endorse using something like a CFAI-style controlled ascent rather than blind FOOM: "A key point in building a young Friendly AI is that when the chaos in the system grows too high (spread and muddle both add to chaos), the Friendly AI does not guess. The young FAI leaves the problem pending and calls a programmer, or suspends, or undergoes a deterministic controlled shutdown."
Useful and interesting list, thanks. I thought the point of defining CEV as what we would choose if we knew better was (partly) that you wouldn't have to subset. We wouldn't be superstitious, vengeful, and so on if we knew better. Also, can you expand on what you mean by "Rawlesian Reflective Equilibrium"? Are you referring (however indirectly) to the "veil of ignorance" concept?
Why not? How does adding factual knowledge get rid of people's desire to hurt someone else out of revenge? People who currently believe in superstitious belief system X would lose the factual falsehoods that X entailed. But most superstitious belief systems have evaluative aspects too, for example, the widespread religious belief that all nonbelievers "ought" to go to hell. I am a nonbeliever. I am also not Chinese, not Indian, not a follower of Sharia Law or Islam, not a member of the Chinese Communist Party, not a member of the Catholic Church, not a Mormon, not a "Good Christian", and I didn't intend to donate all my money and resources to saving lives in the third world before finding out about the singularity. There are lots of humans alive on this planet whose volitions could spring a very nasty surprise on people like us.
Learning about the game-theoretic roots of a desire seems to generally weaken its force, and makes it apparent that one has a choice about whether or not to retain it. I don't know what fraction of people would choose in such a state not to be vengeful, though. (Related: 'hot' and 'cold' motivational states []. CEV seems to naturally privilege cold states, which should tend to reduce vengefulness, though I'm not completely sure this is the right thing to do rather than something like a negotiation between hot and cold subselves.) What it's like to be hurt is also factual knowledge, and seems like it might be extremely motivating towards empathy generally. Why do you think it likely that people would retain that evaluative judgment upon losing the closely coupled beliefs? Far more plausibly, they could retain the general desire to punish violations of conservative social norms, but see above.
I find it interesting that there seems to be a lot of variation in people's views regarding how much coherence there'd be in an extrapolation... You say that choosing a right group of humans is important while I'm under the impression that there is no such problem; basically everyone should be the game, and making higher level considerations about which humans to include is merely an additional source of error. Nevertheless, if there'll be really as much coherence as I think, and I think there'd be hella lot, picking some subset of humanity would pretty much produce a CEV that is very akin to CEVs of other possible human groups. I think that even being an Islamic radical fundamentalist is a petty factor in overall coherence. If I'm correct, Vladimir Nesov has said several times that people can be wrong about their values, and I pretty much agree. Of course, there is an obvious caveat that it's rather shaky to guess what other people's real values might be. Saying "You're wrong about your professed value X, you're real value is along the lines of Y because you cannot possibly diverge that much from the psychological unity of mankind" also risks seeming like claiming excessive moral authority. Still, I think it is a potentially valid argument, depending on the exact nature of X and Y.
And what would you do if Omega told you that the CEV of just {liberal westerners in your age group} is wildly different from the CEV of humanity? What do you think the right thing to do would be then?
3Eliezer Yudkowsky13y
I'd ask Omega, "Which construal of volition are you using?" There's light in us somewhere, a better world inside us somewhere, the question is how to let it out. It's probably more closely akin to the part of us that says "Wouldn't everyone getting their wishes really turn out to be awful?" than the part of us that thinks up cool wishes. And it may even be that Islamic fundamentalists just don't have any note of grace in them at all, that there is no better future written in them anywhere, that every reasonable construal of them ends up with an atheist who still wants others to burn in hell; and if so, the test I cited in the other comment, about filtering portions of the extrapolated volition that wouldn't respect the volition of another who unconditionally respected theirs, seems like it ought to filter that.
I agree that certain limiting factors, tests, etc could be useful. I haven't thought hard enough about this particular proposal to say whether it is really of use. My first thought is that if you have thought about it carefully, then it probably relatively good, just based on your track record.
Eliezer has already talked about this and argued that the right thing would be to run the CEV on the whole of humanity, basing himself partly on an argument that if some particular group (not us) got control of the programming of the AI, we would prefer that they run it on the whole of humanity rather running it on themselves.
1Paul Crowley13y
The lives of most evildoers are of course largely incredibly prosaic, and I find it hard to believe their values in their most prosaic doings are that dissimilar from everyone else around the world doing prosaic things.
I wasn't think of evildoers. I was thinking of people who are just different, and have their own culture, traditions and way of life.
I think that thinking in terms of good and evil belies a closet-realist approach to the problem. In reality, there are different people, with different cultures and biologically determined drives. These cultural and biological factors determine (approximately) a set of traditions, worldviews, ethical principles and moral rules, which can undergo a process of reflective equilibrium to determine a set of consistent preferences over the physical world. We don't know how the reflective equilibrium thing will go, but we know that it could depend upon the set of traditions, ethical principles and moral rules that go into it. If someone is an illiterate devout pentecostal Christian who lives in a village in Angola, the eventual output of the preference formation process applied to them might be very different than if it were applied to the typical LW reader. They're not evil. They just might have a very different "should function" than me.
I think part of the point of what you call "moral anti-realism" is that it frees up words like "evil" so that they can refer to people who have particular kinds of "should function", since there's nothing cosmic that the word could be busy referring to instead. If I had to offer a demonology, I guess I might loosely divide evil minds into: 1) those capable of serious moral reflection but avoiding it, e.g. because they're busy wallowing in negative other-directed emotion, 2) those engaging in serious moral reflection but making cognitive mistakes in doing so, 3) those whose moral reflection genuinely outputs behavior that strongly conflicts with (the extension of) one's own values. I think 1 comes closest to what's traditionally meant by "evil", with 2 being more "misguided" and 3 being more "Lovecraftian". As I understand it, CEV is problematic if most people are "Lovecraftian" but less so if they're merely "evil" or "misguided", and I think you may in general be too quick to assume Lovecraftianity. (ETA: one main reason why I think this is that I don't see many people actually retaining values associated with wrong belief systems when they abandon those belief systems; do you know of many atheists who think atheists or even Christians should burn in hell?)
"One main reason why I think this is that I don't see many people actually retaining values associated with wrong belief systems when they abandon those belief systems; do you know of many atheists who think atheists or even Christians should burn in hell?)" One main reason why you don't see that happening is that the set of beliefs that you consider "right beliefs" is politically influenced, i.e. human beliefs come in certain patterns which are not connected in themselves, but are connected by the custom that people who hold one of the beliefs usually hold the others. For example, I knew a woman (an agnostic) who favored animal rights, and some group on this basis sent her literature asking for her help with pro-abortion activities, namely because this is a typical pattern: People favoring animal rights are more likely to be pro-abortion. But she responded, "Just because I'm against torturing animals doesn't mean I'm in favor of killing babies," evidently quite a logical response, but not in accordance with the usual pattern. In other words, your own values are partly determined by political patterns, and if they weren't (which they wouldn't be under CEV) you might well see people retaining values you dislike when they extrapolate.
Most people may or may not be "Lovecraftian", but why take that risk?
There are gains from cooperating with as many others as possible. Maybe these and other factors outweigh the risk or maybe they don't; the lower the probability and extent of Lovecraftianity, the more likely it is that they do. Anyway, I'm not making any claims about what to do, I'm just saying people probably aren't as Lovecraftian as Roko thinks, which I conclude both from introspection and from the statistics of what moral change we actually see in humans.
I agree that "probability and extent of Lovecraftianity" would be an important consideration if it were a matter of cooperation, and of deciding how many others to cooperate with, but Eliezer's motivation in giving everyone equal weighting in CEV is altruism rather than cooperation. If it were cooperation, then the weights would be adjusted to account for contribution or bargaining power, instead of being equal. To reiterate, "how Lovecraftian" isn't really the issue. Just by positing the possibility that most humans might turn out to be Lovecraftian, you're operating in a meta-ethical framework at odds with Eliezer's, and in which it doesn't make sense to give everyone equal weight in CEV (or at least you'll need a whole other set of arguments to justify that). That aside, the statistics you mention might also be skewed by an anthropic selection effect.
Alternately: They're evil. They have a very different 'should function' to me.
Consider the distinction between whether the output of a preference-aggregation algorithm will be very different for the Angolan Christian, and whether it should be very different. Some preference-aggregation algorithms may just be confused into giving diverging results because of inconsequential distinctions, which would be bad news for everyone, even the "enlightened" westerners. (To be precise, the relevant factual statement is about whether any two same-culture people get preferences visibly closer to each other than any two culturally distant people. It's like with relatively small genetic relevance of skin color, where within-race variation is greater than between-races variation.)
0Paul Crowley13y
I think we agree about this actually - several people's picture of someone with alien values was an Islamic fundamentalist, and they were the "evildoers" I have in mind...
The right thing for me to do is to run CEV on myself, almost by definition. The CEV oracle that I am using to work out my CEV can dereference the dependencies to other CEVs better than I can.
If truly, really wildly different? Obviously, I'd just disassemble them to useful matter via nanobots.
No, not obviously; I can't say I've ever seen anyone else claim to completely condition their concern for other people on the possession of similar reflective preferences. (Or is your point that they probably wouldn't stay people for very long, if given the means to act on their reflective preferences? That wouldn't make it OK to kill them before then, and it would probably constitute undesirable True PD defection ['s_dilemma] to do so afterwards.)
Well, my above reply was a bit tongue-in-cheek. My concern for other things in general is just as complex as my morality and it contains many meta elements such as "I'm willing to modify my preference X in order to conform to your preference Y because I currently care about your utility to a certain extent". On the simplest level, I care for things on a sliding scale that ranges from myself to rocks or Clippy AIs with no functional analogues for human psychology (pain, etc.). Somebody with a literally wildly differing reflective preference would not be a person and, as you say, would be preferably dealt with in True PD manners rather than ordinary human-human altruism contaminated interactions.
This is a very nonstandard usage; personhood is almost universally defined in terms of consciousness and cognitive capacities, and even plausibly relevant desire-like properties like boredom [] don't have much to do with reflective preference/volition.
"If we knew better" is an ambiguous phrase, I probably should have used Eliezer's original: "if we knew more, thought faster, were more the people we wished we were, had grown up farther together". That carries a lot of baggage, at least for me. I don't experience (significant) desires of revenge, so I can only extrapolate from fictional evidence. Say the "someone" in question killed a loved one, and I wanted to hurt them for that. Suppose further that they were no longer able to kill anyone else. Given the time and the means to think about it clearly, I coud see that hurting them would not improve the state of the world for me, or for anyone else, and only impose further unnecessary suffering. The (possibly flawed) assumption of CEV, as I understood it, is that if I could reason flawlessly, non-pathologically about all of my desires and preferences, I would no longer cleave to the self-undermining ones, and what remains would be compatible with the non-self-undermining desires and preferences of the rest of humanity. Caveat: I have read the original CEV document [] but not quite as carefully as maybe I should have, mainly because it carried a "Warning: obsolete" label and I was expecting to come across more recent insights here.
6Wei_Dai13y []
0Paul Crowley13y
I am only part way through but I really recommend that link. So far it's really helped me think about this.
The rest of Rawls' Theory of Justice is good too. I'm trying to figure out for myself (before I finally break down and ask) how CEV compares to the veil of ignorance.
I wish you had written this a few weeks earlier, because it's perfect as a link for the "their associated difficulties and dangers" phrase in my "Complexity of Value != Complexity of Outcome" post. Please consider upgrading this comment to a post, perhaps with some links and additional explanations. For example, what is the ontology problem in ethics?
The ontology problem is the following: your values are defined in terms of a set of concepts. These concepts are essentially predictively useful categorizations in your model of the world. When you do science, you find that your model of the world is wrong, and you build a new model that has a different set of parts. But what do you do with your values?
In practice, I find that this is never a problem. You usually rest your values on some intuitively obvious part whatever originally caused you to create the concepts in question.
Subjective anticipation is a concept that a lot of people rest their axiology on. But it looks like subjective anticipation is an artefact of our cognitive algorithms, and all kinds of Big world theories break it. For example, MW QM means that subjective anticipation is nonsense. Personally, I find this extremely problematic, and in practise, I think that I am just ignoring it.
I think mind copying technology may be a better illustration of the subjective anticipation problem than MW QM, but I agree that it's a good example of the ontology problem. BTW, do you have a reference for where the ontology problem was first stated, in case I need to reference it in the future?
I mentioned it on my blog in august 2008 in the post "ontologies, approximations and fundamentalists" [] Peter de Blanc invented it independently, and I think that one of Eliezer and Marcello probably did too.
2Eliezer Yudkowsky13y
I invented it sometime around the dawn of time, don't know if Marcello did in advance or not. Actually, I don't know if I could have claimed to invent it, there may be science fiction priors.
Thanks for the pointer, but I think the argument you gave in that post is wrong. You argued that an agent smaller than the universe has to represent its goals using an approximate ontology (and therefore would have to later re-phrase its goals relative to more accurate ontologies). But such an agent can represent its goals/preferences in compressed form, instead of using an approximate ontology. With such compressed preferences, it may not have the computational resources to determine with certainty which course of action best satisfies its preferences, but that is just a standard logical uncertainty problem. I think the ontology problem is a real problem, but it may just be a one-time problem, where we or an AI have to translate our fuzzy human preferences into some well-defined form, instead of a problem that all agents must face over and over again.
Yes, if it has compressible preferences, which in reality is the case for e.g. humans and many plausible AIs. In reality problems of the form where you discover that your preferences are stated in terms of an incorrect ontology, e.g. souls, anticipated future experience, are where this really bites. I think that depends upon the structure of reality. Maybe there will be a series of philosophical shocks as severe as the physicality of mental states, Big Worlds, quantum MWI, etc. Suspicion should definitely be directed at what horrors will be unleashed upon a human or AI that discovers a correct theory of quantum gravity. Just as Big World cosmology can erode aggregative consequentialism, maybe the ultimate nature of quantum gravity will entirely erode any rational decision-making; perhaps some kind of ultimate ensemble theory already has. On the other hand, the idea of a one-time shock is also plausible.
The reason I think it can just be a one-time shock is that we can extend our preferences to cover all possible mathematical structures. (I talked about this in Towards a New Decision Theory [].) Then, no matter what kind of universe we turn out to live in, whichever theory of quantum gravity turns out to be correct, the structure of the universe will correspond to some mathematical structure which we will have well-defined preferences over. I addressed this issue a bit in that post as well. Are you not convinced that rational decision-making is possible in Tegmark's Level IV Multiverse?
The next few posts on my blog are going to be basically about approaching this problem (and given the occasion, I may as well commit to writing the first post today). You should read [*] to get a better idea of why I see "preference over all mathematical structures" as a bad call. We can't say what "all mathematical structures" is, any given foundation only covers a portion of what we could invent. As the real world, mathematics that we might someday encounter can only be completely defined by the process of discovery (but if you capture this process, you may need nothing else). -- [*] S. Awodey (2004). `An Answer to Hellman's Question: 'Does Category Theory Provide a Framework for Mathematical Structuralism? []". Philosophia Mathematica 12(1):54-64.
The idea that ethics depends upon one's philosophy of mathematics is intriguing. By the way, I see no post about this on the causality relay!
Hope to finish it today... Though I won't talk about philosophy of mathematics in this sub-series, I'm just going to reduce the ontological confusion about preference and laws of physics to a (still somewhat philosophical, but taking place in a comfortably formal setting) question of static analysis of computer programs.
Great to hear. Looking forward to reading it.
Yes, talking about "preference over all mathematical structures" does gloss over some problems in the philosophy of mathematics, and I am sympathetic to anti-foundationalist views like Awodey's. Also, in general I agree with Roko on the need for an AI that can do philosophy better than any human, so in this thread I was mostly picking a nit with a specific argument that he had. (I was going to remind you about the missing post, but I see Roko already did. :)
I define the following structure: if you take action a, all possible logically possible consequences will follow, i.e. all computable sensory I/O functions, generated by all possible computable changes in the objective physical universe. This holds for all a. This is facilitated by the universe creating infinitely many copies of you every time you take an action, and there being literally no fact of the matter about which one is you. Now if you have already extended your preferences over all possible mathematical structures, you presumably have a preferred action in this case. But the preferred action is really rather unrelated to your life before you made this unsettling discovery. Beings that had different evolved desires (such as seeking status versus maximizing offspring) wouldn't produce systematically different preferences, they'd essentially have to choose at random. If Tegmark Level 4 is, in some sense "true", this hypothetical example is not really so hypothetical - it is very similar to the situation that we are in, with the caveat that you can argue about weightings/priors over mathematical structures, so some consequences get a lower weighting than others, given the prior you chose. My intuition tells me that Level 4 is a mistake, and that there is such a thing as the consequence of my actions. However, mere MW quantum mechanics casts doubt on the idea of anticipated subjective experience, so I am suspicious of my anti-multiverse intuition. Perhaps what we need is the equivalent of a theory of born probabilities for Tegmark Level 4 - something in the region of what Nick Bostrom tried to do in his book on anthropic reasoning (though it looks like Nick simply added more arbitrariness into the mix in the form of reference classes)
I disagree on the first part, and agree on the second part. Yes, and that's enough for rational decision making. I'm not really sure why you're not seeing that...
I agree that you can turn the handle on a particular piece of mathematics that resembles decisionmaking, but some part of me says that you're just playing a game with yourself: you decide that everything exists, then you put a prior over everything, then you act to maximize your utility, weighted by that prior. It is certainly a blow to one's intuition that one can only salvage the ability to act by playing a game of make-believe that some sections of "everything" are "less real" than others, where your real-ness prior is something you had to make up anyway. Others also think that I am just slow on the uptake of this idea. But to me the idea that reality is not fixed but relative to what real-ness prior you decide to pick is extremely ugly. It would mean that the utility of technology to achieve things is merely a shared delusion, that if a theist chose a real-ness prior that assigned high real-ness only to universes where a theistic god existed then he would be correct to pray, etc. Effectively you're saying that the postmodernists were right after all. Now, the fact that I have a negative emotional reaction to this proposal doesn't make it less true, of course.
There is a deep analogy between how you can't change the laws of physics (contents of reality, apart from lawfully acting) and how you can't change your own program. It's not a delusion unless it can be reached by mistake. The theist can't be right to act as if a deity exists unless his program (brain) is such that it is the correct way to act, and he can't change his mind for it to become right, because it's impossible to change one's program, only act according to it.
The problem is that this point of view means that in a debate with someone who is firmly religious, not only is the religious person right, but you regret the fact that you are "rational"; you lament "if only I had been brought up with religious indoctrination, I would correctly believe that I am going to heaven". Any rational theory that leaves you lamenting your own rationality deserves some serious scepticism.
Following the same analogy, you can translate it as "if only the God did in fact exist, ...". The difference doesn't seem particularly significant -- both "what ifs" are equally impossible. "Regretting rationality" is on a different level -- rationality in the relevant sense is a matter of choice. The program that defines your decision-making algorithm isn't. I still fear that you are reading in my words something very different from what I intend, as I don't see the possibility of a religious person's mind actually acting as if God is real. A religious person may have a free-floating network of beliefs about God, but it doesn't survive under reflection. A true god-impressed mind would actually act as if God is real, no matter what, it won't be deconvertable, and indeed under reflection an atheist god-impressed mind will correctly discard atheism. Not all beliefs are equal, a human atheist is correct not just according to atheist's standard, and a human theist is incorrect not just to atheist's standard. The standard is in the world, or, under this analogy, in the mind. (The mind is a better place for ontology, because preference is also here, and human mind can be completely formalized, unlike the unknown laws of physics. By the way, the first post is up []).
So your argument is that the reason that the theists are wrong is because they only sorta-kinda believe in God anyway, but if they really believed, then they'd be just as right as we are?
But only in the sense that their calculation could be correct according to a particularly weird prior []. The difference between normal theist and a "god-impressed mind" who both believe in God is that of rationality: the former makes mistakes in updating beliefs, the latter probably doesn't. The same with an atheist god-impressed mind and a human atheist. You can't expect to find that weird a prior in a human. And of course, you should say that the god-impressed are wrong about their beliefs, though they correctly follow the evidence according to their prior. If you value their success in the real world more than the autonomy of their preference, you may want to reach into their minds and make appropriate changes. I should say again: the program that defines the decision-making algorithm can't be normally changed, which means that one can't be really "converted" to a different preference, though one can be converted to different beliefs and feelings. Observations don't change the algorithm, they are processed according to that algorithm. This means that if you care about reflective consistency (and everyone does, in the sense of preservation of preference), you'd try to counteract the unwanted effects of environment on yourself, including the self-promoting effects where you start liking the new situation. The extent to which you like the new situation, the "level of conviction", it's pretty much irrelevant, just as the presence of a losing psychological drive []. It'd take great integrity (not "strength of conviction") in the change for significantly different values to really sink in, in the sense that the new preference-on-reflection will resemble the new beliefs and feelings similarly to how the native preference-on-reflection will resemble native (sane, secular, etc.) beliefs and feelings.
I doubt that you can define a way to choose an algorithm out of a human brain that makes that sentence true.
Yes, that wasn't careful. In this context, I mean "no large shift of preference". Tiny changes occur all the time (and are actually very important if you scale them up by giving the preference with/without these changes to a FAI). You can model the extent of reversibility (as compared to a formal computer program) by roughly what can be inferred about the person's past, which doesn't necessarily all has to be from the person's brain. (By an algorithm in human brain I mean all of human brain, basically a program that would run an upload implementation, together with the data.)
I agree that it's ugly to think of the weights as a pretense on how real certain parts of reality are. That's why I think it may be better to think of them as representing how much you care about various parts of reality. (For the benefit of other readers, I talked about this in What Are Probabilities, Anyway? [].) Actually, I haven't completely given up the idea that there is some objective notion of how real, or how important, various parts of reality are. It's hard to escape the intuition that some parts of math are just easier to reach or find than others, in a way that is not dependent how human minds work.
I believe even personal identity falls under this category. A lot of moral intuitions work with the-me-in-the-future object, as marked in the map. To follow these intuitions, it is very important for us to have a good idea of where the-me-in-the-future is, to have a good map of this thing. When you get to weird thought experiments with copying, this epistemic step breaks down, because if there are multiple future-copies, the-me-in-the-future is a pattern that is absent. As a result, moral intuitions, that indirectly work through this mark on the map, get confused and start giving the wrong answers as well. This can be readily observed for example from preferential inconsistency in time expected in such thought experiments (you precommit to teleporting-with-delay, but then your copy that is to be destroyed starts complaining). Personal identity is (in general) a wrong epistemic question asked by our moral intuition. Only if preference is expressed in terms of the territory (or rather in a form flexible enough to follow all possible developments), including the parts currently represented in moral intuition in terms of the-me-in-the-future object in the territory, will the confusion with expectations and anthropic thought experiments go away.
* thanks, I'll consider it
2Paul Crowley13y
Isn't this one of the problems you can let the FAI solve?
Actually the repugnant conclusion yes, preference aggregation no, because you have to aggregate individual humans' preferences.
And what if preferences cannot be measured by a common "ruler"? What then?
I agree that preference aggregation is hard. Wei dai and nick Bostrom have both made proposals based upon agents negotiating with some deadline or constraint.
Maybe I'm crazy but all that doesn't sound so hard. More precisely, there's one part, the solution to which should require nothing more than steady hard work, and another part which is so nebulous that even the problems are still fuzzy. The first part - requiring just steady hard work - is everything that can be reduced to existing physics and mathematics. We're supposed to take the human brain as input and get a human-friendly AI as output. The human brain is a decision-making system; it's a genetically encoded decision architecture or decision architecture schema, with the parameters of the schema being set in the individual by genetic or environmental contingencies. CEV is all about answering the question: If a superintelligence appeared in our midst, what would the human race want its decision architecture to be, if we had time enough to think things through and arrive at a stable answer? So it boils down to asking, if you had a number of instances of the specific decision architecture human brain, and they were asked to choose a decision architecture for an entity of arbitrarily high intelligence that was to be introduced into their environment, what would be their asymptotically stable preference? That just doesn't sound like a mindbogglingly difficult problem. It's certainly a question that should be answerable for much simpler classes of decision architecture. So it seems to me that the main challenge is simply to understand what the human decision architecture is. And again, that shouldn't be beyond us at all. The human genome is completely sequenced, we know the physics of the brain down to nucleons, there's only a finite number of cell types in the body - yes it's complicated, but it's really just a matter of sticking with the problem. (Or would be, if there was no time factor. But how to do all this quickly is a separate problem.) So to sum up, all we need to do is to solve the decision theory problem 'if agents X, Y, Z... get to determine the value
Could you clarify for me what you mean by requiring that that a human consciousness be instantiated? Is it that you don't believe it is possible to elicit a CEV from a human if instantiation is involved or that you object to the consequences of simulating human consciousnesses in potentially undesirable situations? In the case of the latter I observe that this is only a problem under certain CEVs and so is somewhat different in nature to the other requirements. Some people's CEVs could then be extracted more easily than others.
I am no where near caught up on FAI readings but here are is a humble thought. What I have read so far seems to be assuming a single jump FAI. That is once the FAI is set it must take us to where we ultimately want to go without further human input. Please correct me if I am wrong. What about a multistage approach? The problem that people might immediately bring up is that a multistage approach might lead elevating subgoals to goals. We say, "take us to mastery of nanotech" and the AI decides to rip us apart and organize all existing ribosomes under a coherent command. However, perhaps what we need to do is verify that any intermediate state goal better than the current state. So what if we have the AI guess a goal state. Then simulate that goal state and expose some subset of humans to that simulation. The AI the asks "Proceed to this stage or no" The humans answer. Once in the next stage we can reassess. To give a sense of motivation: it seems that verifying the goodness of future-state is easier than trying to construct the basic rules of good statedness.
Someone doesn't like Bach because he was traumatized by an exposure to classical music at a tender age? Give me a break. Music is like languages, not math -- the surest way to learn to like Bach is full immersion at a young age, not a graduated curriculum that starts from "lower" forms of music.
That's a very nice story.

I made a couple posts in the past that I really hoped to get replies to, and yet not only did I get no replies, I got no karma in either direction. So I was hoping that someone would answer me, or at least explain the deafening silence.

This one isn't a question, but I'd like to know if there are holes in my reasoning.

Here, I had a question:

I looked at your consciousness comment. First, consciousness is notoriously difficult to write about in a way that readers find both profound and comprehensible. So you shouldn't take it too badly that your comment didn't catch fire.

Speaking for myself, I didn't find your comment profound (or I failed to comprehend that there was profundity there). You summarize your thesis by writing "Basically, a qualium is what the algorithm feels like from the inside for a self-aware machine." (The singular of "qualia" is "quale", not "qualium", btw.)

The problem is that this is more like a definition of "quale" than an explanation. People find qualia mysterious when they ask themselves why some algorithms "feel like" anything from the inside. The intuition is that you have both

  1. the code — that is, an implementable description of the algorithm; and

  2. the quale — that is, what it feels like to be an implementation of the algorithm.

But the quale doesn't seem to be anywhere in the code, so where does it come from? And, if the quale is not in the code, then why does the code give rise to that quale, rather than to some other one?

T... (read more)

"So you shouldn't take it too badly that your comment didn't catch fire." I'm not mad, but... Just see it from my point of view. An interesting thought doesn't come to guys like me every day. ;) "But the quail doesn't seem to be anywhere in the code, so where does it come from?" I think it's in the code. When I try to imagine a mind that has no qualia, I imagine something quite unlike myself. What would it actually be like for us to not have qualia? It could mean that I would look at a red object and think, "object, rectangular, apparent area 1 degree by 0.5 degrees, long side vertical, top left at (100, 78), color 0xff0000". That would be the case where the algorithm has no inside, so it doesn't need to feel like anything from the inside. Nothing about our thoughts would be "ineffable". (Although it would be insulting to call a being unconscious or, worse, "not self aware" for knowing itself better than we do... Hmm. I guess qualia and consciousness are separate after all. Or is it? But I'm dealing with qualia right now.) Or, the nerve could send its impulse directly into a muscle, like in jellyfish. That would mean that the hole in my knowledge is so big that the quail for "touch" falls through it. In my mind, touch leaves a memory, and I then try to look at this memory. I ask my brain, "what does touch feel like?", and I get back, "Error: can't decompile native method. But I can tell you definitely what it doesn't feel like: greenness." So what I'm saying is, I can't observe what the feeling of touch is made of, but it has enough bits to not confuse it with green. It makes me [feel] unconfused. Although it might be confusing. "Just to be clear, I think that those questions arise out of a wrong approach to consciousness." What's your approach?
I don't understand your explanation. You are apparently saying that quale (you seem to deliberately misspell [] the word, why?) is how the algorithm feels from inside. Well, I agree, but in the same time I think that "quale" is only a philosopher's noble word for "feel from inside". The explanation looks like a truism. I have always been (and still am) confused by questions like: How other people perceive colors? Do they feel it the same way as I do? Are there people who see the colors inverted, having the equivalent of my feeling of "redness" when they look at green objects? They will call that feeling "greenness", of course, but can their redness be my greenness and vice versa? What about colorblind people? If I lost the ability to recognise blue from green, would I feel blueness or greenness when looking at those colors? What does it in fact mean, to compare feelings of different people? Or even, how does it feel to be a dog? A fish? A snail? I am almost sure that the questions themselves are confused, without clear meaning, and can be explained away as such, but still I find them appealing in some strange way. I always wanted to make an experiment on myself, but I am also afraid of it and don't have an opportunity. I would buy glasses which invert colors like on a photographic negative and wear them without interruption for some time. Certainly in the beginning I would feel redness when looking at trees, but it can be that I would accomodate and start feeling greenness instead. Or not. Certainly, it would be a valuable experinence. Has anybody tried something similar?
I once heard of people at UCSD who had plans to experiment with inverted spectrum goggles. Jonathan Cohen [] would know more.
Thanks, seems interesting.
Would this be possible optically? The only way I can see it working is using a live video feed with some image processing to invert colours. That is probably quite practical using modern technology though.
Interesting experiment. It reminds me of an experiment where subjects wore glasses that turned world upside down (really, right side up for the projection on our eye) and eventually they adjusted so the world looked upside down when taking off the glasses. What do you think a "yes" or "no" in your experiment would mean? Note, Dennett says in Quining Qualia [] :
I know about the experiment you mention, and it partly motivated my suggestion; I just subjectively find "yellowness" and "blueness" more qualious than "upness" or "leftness". In my experiment, "yes" would mean that there would be no dissonance between memories and perceptions, that I would just not feel that the trees are red or purple, but green, and find the world "normal". That I would, one day, cease to feel the need to get rid of the color-changing glasses, and my aesthetic preferences would remain the same as they were in the pre-glasses period. I think it's likely - based on the other subjects' experiences with upside-down glasses - that it would happen after a while, but the experience itself may be more interesting than the sole yes/no result, because it is undescribable. That's one problem with qualia: they are outside the realm of things which can be described. Describing qualia is like describing flavour of an unknown exotic fruit: no matter how much you try, other people wouldn't understand until they degust it themselves.
With regards to Karma, it could be equal up and down. This is less likely if you were checking it. Digg allows you to view plus and minus karma counts separately, for one.

I just failed the Wason selection task. Does anyone know any other similarly devilish problems?

Here's a classic: Linda is 31 years old, single, outspoken, and very bright. She majored in philosophy. As a student, she was deeply concerned with issues of discrimination and social justice, and also participated in anti-nuclear demonstrations. Which is more probable? 1. Linda is a bank teller. 2. Linda is a bank teller and is active in the feminist movement. Answer here [].
Fun task. I'll second the request.

What's an easy way to explain the paperclip thing?

We happen to like things like ice cream and happiness. But we could have liked paperclips. We could have liked them a lot, and not liked anything else enough to have it instead of paperclips. If that had happened, we'd want to turn everything into paperclips - even ourselves and each other!

Oh, look honey: more proof wine tasting is a crock:

A French court has convicted 12 local winemakers of passing off cheap merlot and shiraz as more expensive pinot noir and selling it to undiscerning Americans, including E&J Gallo, one of the United States' top wineries.

Cue the folks claiming they can really tell the difference...

There's plenty of hard evidence that people are vulnerable to priming effects and other biases when tasting wine. There's also plenty of hard evidence that people can tell the difference between wine A and wine B, under controlled (blinded) conditions. Note that "tell the difference" isn't the same as "identify which would be preferred by experts". So, while the link is factually interesting, and evidence that some large-scale deception is going on, aided by such priming effects as label, marketing campaigns and popular movies can have, it seems a stretch to call it "proof" that people in general can't tell wine A from wine B. Rather, this strikes me as a combination of trolling and boo lights: cheaply testing who appears to be "on your side" in a pet controversy. How well do you expect that to work out for you, in the sense of "reliably entangling your beliefs with reality"?
I think I'm entangling my beliefs with reality very well, by virtue of extracting all available information from phenomena rather than retreat to evidence that agrees with me. (Let's not forget, I didn't start out thinking that it was all BS.) For example, did you stop to notice the implications of this: How does that compare to the priming effects for other drinks? Does it matter? But what would be the appropriate comparison? They were passing of as expensive, something that's actually cheap. Where else would that work so easily, for so long? Normally, if you tried that, it would be noticed quickly, if not immediately, by virtually everyone. What if you tried to pass off 16 oz of milk as 128? Or spoiled milk as milk expiring in a week? Then, factor in how much difference is claimed to exist in wine vs. milks. Who's optimally using evidence here?
Art forgeries. (Which shows that the value of the painting is determined by the status of the artist and not the quality of the art.) If I can paint a painting that convinces experts that it was painted by [insert expert painter here], does that mean I'm as good an artist as said painter? (Assuming that my painting isn't a literal copy of someone else's.)
Which, like wine, is another example of a path-dependent collective delusion that's not Truly Part of [] our values. (That is, our valuation of the work wouldn't survive deletion of the history that led to such a valuation.) Very nearly yes, it does, modulo a few factors. If you produced it after the artist, then you are benefiting from the artist's already having identified a region of conceptspace that you did not find yourself. (If the art is revered because of the artist's social status, that it wasn't even much of an accomplishment to begin with.) To put it another way, you produced the work after "supervised learning", while the artist didn't need that particular training. If you can pass off a previous work of yours as being one of the artist's, that definitely makes you better.
Who is "we", here? The problem I have is not that you're wrong, for the people you're talking about; it's that you (probably) overestimate the size and/or importance of that population. You're not telling the whole truth, in effect. There are plenty of people who like paintings for the way they look, and would happily buy the work of a lesser-known artist at a cheap price if they liked it. Yes, some people use art to status-signal, but some people also actually like art. (There may even be a nonempty intersection!)
Sorry if I sound dodgy here, but I don't think I've said anything that contradicts this. My criticism is of these two things: 1) the idea that the elite-designated "high art" is non-arbitrary. (I claim it's a status-reinforced information cascade that wouldn't regain the designation of high-art if you deleted knowledge of which ones had been so classified.) 2) the excessive premiums paid for artworks based on both 1) and the fact that they are the originals (a "piece of history"). Never have I criticized or denied the existence of people who buy artworks because they simply like it and it appeals to them. I just criticize the way that we're expected to agree with the laurels attached to elite-designated high art. As I said before, I would have no problem if art were just a matter of "hey, I like this, now get on with your lives" (as it works in e.g. video games).
Often the worth of an artist stems from inventing new possibilities. Copycats are lesser.
You seem to want a contest. The other option, where we are both "on the side of truth", appeals to me more. We're fortunate in having different experiences in the domain of taste. I'm one of those people who like wine, and I'm confident I can identify some of its taste characteristics in blind tests. So, predictably I resent language which implies I'm an idiot, but I'm open to inquiry. Our investigation should probably begin "at the crime scene", that is, close to what evidence we can gather about the sense of taste. So, yes, we could examine similar priming effects on other drinks. I have a candidate in mind, but what I'd like to ask you first is, suppose I name the drink I have in mind and we then go look for evidence of fraud in its commerce. What would it count as evidence of if we found no fraud? If we did find it? Which one would you say counts as evidence that "people can't tell the difference" between wines?
You can easily test yourself if you have a confederate. I recommend a triangle test [].
If someone was long ago made aware of powerful evidence that an expensive pleasure he currently enjoys can be replicated with fidelity at a sliver of the cost, and he hasn't already done the experimentation necessary to properly rule this out, then you're right. There are explanations other than that he is an idiot. But they're not much more flattering either. I can tell you that if I were in this position for another beverage, I would have already done the tests. The no-fraud case is positive but weak evidence of for people telling the difference because it can be accounted for by honesty on the part of retailers, fear of whistleblowers, etc. Finding fraud is unlikely but strong evidence against the claim that people can tell the difference -- because it crucially depends on the priming effects (or some other not-truly-part-of-you effect) dominating. I'd prefer a blind comparison on the cheap substitutes like Cyan suggested. And I'd be glad that you've identified a product I'm overpaying for!
What would you suggest I do, to replicate the pleasure I get from wine?
You mean, replicate the pleasure from expensive wine? (I'm going to assume you genuinely like the act of drinking wine.) Easy: accept that it's an illusion, then buy the cheap stuff (modulo social status penalties) and prime it the way good wines are. (This may require an assistant.) Gradually train yourself to regard them as the same quality. If you can trained to put it on a pedestal, you can probably be trained to take it off. If it's unavoidable that you discount the taste due to your residual knowledge that the wine is low-class, then accept that you're overpaying because of a unremovable bias. (Not a slight against you, btw -- I admit I was the same way with CFLs for a while, in that I couldn't discard the knowledge, and thus negative affect, that they're CFLs rather than incandescent, and thereby pointlessly overpaid for lighting.) On the more realistic assumption that you've simply been trained to like wine via a process that would equally well train you to like anything, get some friends together and train yourselves to like V8. (The veggie drink, not the engine.)
Wait, can you expand on how fraud in the trade of a non-alcoholic drink is "strong but unlikely" evidence that people cannot tell the differences between wines? Such fraud might be evidence that people cannot tell the difference between tastes more generally, but that seems like a higher hurdle to clear.
Where did I say that? The claim you quoted from me was about "what would be evidence of people's ability to distinguish that non-alcoholic drink", not wines. You'd check for people's ability to distinguish wines by fraud in wines, and people's ability to distinguish specific non-wine drinks by fraud in specific non-wine drinks. I thought I made that very clear, and if not, common sense and the principle of charity should have sufficed for you not to infer that I would get the "wires crossed" like that. (ETA: By the way: is there a shorter way to refer to a person's ability to distinguish foods/drinks? I've tried shorter expressions, but they make other posters go batty about the imprecision without even suggesting an alternative. Paul Birch suggests [] "taste entropy of choice", but that's obscure.) And, to preempt a possible point you may be trying to make: yes, fruit drinks may be fraudulently labeled as real fruit juice, but that's not a parallel case unless people claim to be able to distinguish by taste the presence of real juice and purchase on that basis.
So we've averted misunderstanding, good. My question remains: what does fraud (or non-fraud) in non-alcoholic drinks tell us about whether people "really can tell the difference" between wines? Just to be sure what you're claiming, btw: if I did a "triangle test", blinded, on two arbitrary bottles of wine from my supermarket, and I could tell them apart, would you retract that claim? Or is your claim restricted to some specific varietals?
How could I have explained my position better so that you would not have inferred the point about fruit drinks? It doesn't remain. If people can tell the difference, you don't gain from fraud. If they can't, you could gain from fraud. Where's the confusion? I accept that you could tell red from white, so it couldn't be completely random. I'd want a test over the two varieties they said were swapped in the story, or within the same variety, but significant cost difference.
I have inferred nothing about fruit drinks. In this comment [] you replied to me with an allusion to "other drinks". Later in the same comment you referred to milk. In other words, you primed me to think about non-alcoholic drinks. Later in the exchange, you ruled out the possibility that fraud in non-alcoholic drinks will provide any evidence relevant to your original claim. So we're better off dropping this line of inquiry altogether. The story mentioned three varietals: merlot, shiraz and pinot noir. It seems likely that in the next few days I will be able to procure half-bottles of a Merlot (vin de pays, 2005) at €6.5/l and a Pinot Noir (Burgundy, 2005) at 14€/l. The experiment will set me back about 10€. Do you consider the experiment valid if I buy the bottles myself, and have a third person prepare the glasses? Would you care to stipulate any particular controls? Are you willing to trust my word when I report back with the results? Do I have to correctly identify the cheaper and the more expensive wine, or just to show I can tell the difference? What will you bet on the outcome?
There's a difficulty that affects tests like these: you can be much more sensitive when you know you're being tested, so it wouldn't tell as much about people who can be fooled when given wine that they didn't expect something was wrong with. The test would have to be made slightly more difficult in order to account for your preparation. Here's what I would consider a fair test: I pick three wines, (or imitations of wines) and then do a modified version of the triangle test: I can make all three the same, all three different, or just two the same, and not tell you which. To pass, you have to correctly rank their prices and identify which, if any, are the same. (You'd be allowed to count two different ones as effectively the same prices if they differed only by a few Euros.) And of course, the test would have to be triple blind, etc.
Why thank you. So, you are expecting that in some situations people actually can tell wines apart? Sounds complex. I might play, but you'd have a) to pick three wines I can actually pick up in my local supermarket chain, Monoprix; they have an online catalog; b) to write the instructions with enough precision that a local confederate can carry them out. I'm willing to do the work myself for the simple case (have done so in the grandparent), but not if the rules get too baroque. There's one thing I object to. Why do I have to rank them by price? I am not claiming that the more expensive wines are systematically better, taste-wise, than the cheaper ones. I do claim that the very good stuff is more likely to be found in the more expensive bottles. There is no reason to expect that price will be systematically correlated with my idiosyncratic, relatively untrainted tastes. If I have to rank them by price via recognition, I'd have to have tried them out beforehand. If your hypothesis is correct that shouldn't affect the results, since you're claiming my discrimination of "better" comes from priming effects.
Sure, just like how, if you put enough effort in, you can tell any two drinks apart. The difference is that people make these claims about wine without have done the exercises necessary for precise discernment, yet claim these subtleties matter to them. You're really not grasping the concept of biasproofing, are you? "A confederate" taints the experiment. Your purchasing the wines taints the experiment. Okay, well, the wine cheerleaders do, so this is one way you break from them and agree with me. Does not follow: if you're told in advance which has the label "good", you can spit that information back out latter. My claim is that a judgment of good dependent on being told in advance that it's good is not a genuine judgment of good.
Not performing the experiment taints it even more. But perhaps by now we've learned enough of each other's claims. I leave it to you to find a way forward.
If "top winery" means "largest winery", as it does in this story, I don't see how it says anything about the ability of tasters to tell the difference. Those who made such claims probably weren't drinking Gallo in the first place. I think it's closer to say they were passing off as cheap, something that's actually even cheaper. Switch the food item and see if your criticism holds: Wonderbread, America's top bread maker, was conned into selling inferior bread. So-called "gourmets" never noticed the difference! Bread tasting is a crock.
If people made such a huge deal about the nuances in the taste of bread, while it also "happened" to have psychoactive effects that, gosh, always have to be present for the bread to be "good enough" for them, and cheap breads were still normally several times the cost of comparable-nutrition food, then yes, the cases would be parallel. (Before anyone says it: Yes, I know bread as trace quantities of alcohol, we're all proud of what you learned in chemistry.)
If people who can tell the difference are a big enough demographic to sell to, then they are employed by all wineries, regardless of quality. But an alternate explanation is that Gallo was tacitly in on the scam - they got as much PN as Sideways demanded, without moving the market.
Ah, I misunderstood the comment. I just assumed that Gallo was in on it, and the claim was that customers of Gallo failing to complain constituted evidence of wine tasting's crockitude. If Gallo's wine experts really did get taken in, then yes, that's pretty strong evidence. And being the largest winery, I'm sure they have many experts checking their wines regularly--too many to realistically be "in" on such a scam. So you've convinced me. Wine tasting is a crock.
I love this kind of thing. Shall we name this phenomenon the "Emperor's New Clothes Effect"?
That could be a general name for the phenomenon. As it relates to wine tasting (and maybe we could stretch it a bit), I'd propose "the Nuances of Toast Effect", for a particularly memorable phrase in this Dave Barry column [].
I can't even tell the difference between Coke and Pepsi.
One's in a red can, the other in a blue one ;-) oh well, me neither actually.
Really? How much soda do you drink? The difference is readily noted by me. I can even tell the difference between regular and diet Coke.
This seems to suggest that it is easier to tell the difference between Coke and Pepsi than it is to tell the difference between regular and diet. I can tell the difference between all of them, but the first is a lot harder and I think that experience is pretty common. Most diet drinks use aspartame as a sugar substitute and aspartame leaves a very distinctive aftertaste in my mouth.
Wait, is that supposed to be harder? I'm not sure I could tell the difference between Coke and Pepsi, but I think I could tell the difference between regular and diet.
I don't like cola very much, so if you gave me a drink of fizzy black stuff, I wouldn't be able to identify which brand it was. Also, other sodas have a tendency to give me heartburn so I've been drinking them much less than I used to.
Is this unusual?
I have heard people remark they can't tell the difference. My father for example.

Nice recap of psychological biases from the Charlie Munger school (of hard knocks and making a billion dollars).

I've been wondering what the existance of Gene Networks tells us about recursively self improving systems. Edit: Not that self-modifying gene networks are RSIS, but the question is "Why aren't they?" In the same way that failed attempts at flying machines tell us something, but not much, about what flying machines are not. End Edit

They are the equivalent of logic gates and have the potential for self-modification and reflection, what with DNAs ability to make enzymes that chop itself up and do so selectively.

So you can possibly use them as evide... (read more)

It seems like in gene networks, every logic gate has to evolve separately, and those restriction enzymes you mention barely do anything but destroy foreign DNA. That's less self-modification potential than the human brain.
The inability to create new logic gates is what I meant by the systems having low memory. In this case low memory to store programs. Restriction enzymes also have a role in the insertion of plasmids [] into genes. An interesting question is: If I told you about computer model of evolution with things like plasmids, controlled mutation [\(D\]J_recombination); would you expect it to be potentially dangerous? I'm asking this to try to improve our thinking about what is and isn't dangerous. To try and improve upon the kneejerk "everything we don't understand is dangerous" opinion that you have seen.
Well, I'm not familiar enough with controlled mutation to be able to say anything useful about it.
2Eliezer Yudkowsky13y
I wonder if the distinction between self-modification and recursive self-improvement is one of those things that requires a magic gear to get, and otherwise can't be explained by any amount of effort.
I understand there is a distinction. Would you agree that RSI systems are conceptually a subset of self-modifying (SM) systems? One that we don't understand what exact properties make a SM system one that will RSI. Could you theoretically say why EURISKO didn't RSI? I was interested in how big a subset. The bigger it is the more dangerous, the more easily we will find it.
Sure. In fact, some of the Lenat quotes on LW even tell you why. As a hack to defeat 'parasitic' heuristics, Lent (& co.?) put into Eurisko a 'protected kernel' which couldn't be modified. This core was not good enough to get everything going, dooming Eurisko from a seed AI perspective, and the heuristics never got anywhere near the point they could bypass the kernel. Eurisko was inherently self-limited.
It seems to me that for SM to become RSI, the SM has to able to improve all the parts of the system that are used for SM, without leaving any "weak links" to slow things down. Then the question is (slightly) narrowed to what exactly is required to have SM that can improve all the needed parts.
Does my edit make more sense now?
2Eliezer Yudkowsky13y
Sure, but the answer is very simple. Gene regulatory networks are not RSI because they are not optimization processes.
Intrinsically they aren't optimization processes but they seem computationally expressive enough for an optimization process to be implemented on them (the same way X86-arch computers aren't optimization processes). And if you are a bacteria it seems it should be something that is evolutionarily beneficial, so I wouldn't be surprised to find some optimization going on at the gene network level. Enough to be considered a full optimization process I don't know, but if not, why not?
3Eliezer Yudkowsky13y
But they aren't optimization processes. It doesn't matter if they could implement one, they don't. You might as well point to any X86 chip and ask why it doesn't RSI.
I'm not talking about any specific Gene Network, I'm talking about the number and variety of gene networks that have been explored through out evolutionary history. Do you know that they all aren't optimisation processes? That they haven't popped up at least once? To my mind it is asking why a very very large number of simple x86 systems (not just chips, they have storage) each with a different program that you don't know the details of hasn't RSId. Which I don't think is unreasonable. How many distinct bacterial genomes do you think there has been since the beginning of life? Considering people estimate 10 million+ bacterial species alive today. Some people have talked about the possibility of brute forcing AGI through evolutionary means, I'm simply looking at a previous evolutionary search through computational system space to get some clues.
A gene network optimizes the use of resources to make more copies of that gene network. It sense the environment, and its own operations, and adjusts what it is doing in response. I think it is an optimization process.
Evolution is stupid and optimization processes are complicated. Do you not think that's an adequate explanation?
The question is, Why did evolution get thus far and no further? Can you give an account that simultaneously explains both of the observed bounds? I suppose that some would be happy with "Shear difficulty explains why evolution did no better, and anthropics explains why it did no worse." But I don't find that especially satisfying.
Evolution managed to make an optimisation process in our heads, but not one in anything's genes. It had had a lot more time to work with genes as well. Why? It is possibly worth noting that I am not talking about optimising proteins but the network that controls the activation of the genes. Protein folding is hard.
It may be that getting optimization into our heads was the easiest way to get it into our genes (eventually, when we master genetic engineering).
0Paul Crowley13y
Possibly, but if you could link to your best efforts to explain it I'd be interested. I tried Google []... EDIT: D'oh! Thanks Cyan!
Shoulda tried the Google custom search bar: Recursive self-improvement [].
You're just lucky there's no such thing as LMG(CSB)TFY. ;-)
Such things probably happen because effort spent on explaining quickly hits diminishing returns if the other person spends no effort on understanding.
They already are RSIS, if you believe in the evolution of evolvability, which you probably should. The probable evolution of DNA from RNA, of introns, and of sex are examples of the evolution of evolvability. There are single-celled organisms that act intelligently despite not having (or being) neurons. The slime mold, for example. A gene network is a lot like the brain of an insect in which the exact connectivity of every neuron is predetermined. However, its switching frequency is much slower. More advanced brains have algorithms that can use homogenous networks. That means that you can simply increase the number of neurons made, and automatically get more intelligence out of them. Organisms have 600 to 20,000 genes. A honeybee has about a million neurons.
What does "intelligence" mean here? For context, some more about slime moulds []. In that thesis is a detailed model of the whole life-cycle of the slime mould, using biochemical investigations and computer modelling to show how all the different stages and the transitions between them happen. What does it mean, to say that this system is "intelligent"? The word is used for a very wide range of things, from slime moulds (and perhaps even simpler systems?) to people and beyond. What is being claimed when the same word is applied to all of these things? Put in practical terms, does a detailed knowledge of exactly how the slime mould works help in constructing an AGI? Does it help in constructing more limited sorts of AI? Does it illuminate the investigation of other natural systems that fall within the concept of "intelligence"? I am not seeing a reason to answer "yes" to any of these questions.
Yes, to all of those questions. I don't think we currently have the AI technology needed to produce something with the intelligence of a slime mold. (Yes, we might be able to, if we gave it magical sensors and effectors, so that it just had to say "go this way" or "go that way". Remember that the slime mold has to do all this by directing an extremely complex sequence of modifications to its cytoskeleton.) Therefore, having a detailed knowledge of how it did this, and the ability to replicate it, would advance AI.

Suppose I wanted to convince someone that signing up for cryonics was a good idea, but I had little confidence in my ability to persuade them in a face-to-face conversation (or didn't want to drag another discussion too far off-topic) - what is the one link you would give someone that is most likely to save their life? I find the pro-cryonics arguments given by Eliezer and others on this site + Overcomi