All of wuwei's Comments + Replies

Hacking the CEV for Fun and Profit

CEV is not preference utilitarianism, or any other first-order ethical theory. Rather, preference utilitarianism is the sort of thing that might be CEV's output.

6Wei_Dai12yObviously CEV isn't identical to preference utilitarianism, but CEV and preference utilitarianism have the following principles in common, which the hack exploits: * Give people what they want, instead of what you think is good for them. * If different people want different things, give each individual equal weight. It seems clear that Eliezer got these ideas from preference utilitarianism, and they share some of the same flaws as a result.
Human values differ as much as values can differ

Matt Simpson was talking about people who have in fact reflected on their values a lot. Why did you switch to talking about people who think they have reflected a lot?

What "someone actually values" or what their "terminal values" are seems to be ambiguous in this discussion. On one reading, it just means what motivates someone the most. In that case, your claims are pretty plausible.

On the other reading, which seems more relevant in this thread and the original comment, it means the terminal values someone should act on, which we might ... (read more)

Only humans can have human values

I suppose I might count as someone who favors "organismal" preferences over confusing the metaphorical "preferences" of our genes with those of the individual. I think your argument against this is pretty weak.

You claim that favoring the "organismal" over the "evolutionary" fails to accurately identify our values in four cases, but I fail to see any problem with these cases.

  • I find no problem with upholding the human preference for foods which taste fatty, sugary and salty. (Note that consistently applied, the "
... (read more)
3PhilGoetz12yVoted up for thought and effort. BTW, when I started writing this last week, I thought I always preferred organismal preferences. That's a good point. But in the context of designing a Friendly AI that implements human values, it means we have to design the AI to like fatty, sugary, and salty tastes. Doesn't that seem odd to you? Maybe not the sort of thing we should be fighting to preserve? I don't see how. Are you going to kill the snakes, or not? Do you mean that you can use technology to let people experience simulated violence without actually hurting anybody? Doesn't that seem like building an inconsistency into your utopia? Wouldn't having a large number of such inconsistencies make utopia unstable, or lacking in integrity? That's how I said we resolve all of these cases. Only it doesn't get outweighed by a single different value (the Prime Mover model); it gets outweighed by an entire, consistent, locally-optimal energy-minimizing set of values. This seems to be at the core of your comment, but I can't parse that sentence. My emphasis is not on defeating opposing views (except the initial "preferences are propositions" / ethics-as-geometry view), but on setting out my view, and overcoming the objections to it that I came up with. For instance, when I talked about the intuitions of humans over time not being consistent, I wasn't attacking the view that human values are universal. I was overcoming the objection that we must have an algorithm for choosing evolutionary or organismal preferences, if we seem to agree on the right conclusion in most cases. Which conclusion did you have in mind? The key conclusion is that value can't be unambiguously analyzed at a finer level of detail than the behavior, in the way that communication can't be unambiguously analyzed at a finer level of detail than the proposition. You haven't said anything about that. (I just realized this makes me a structuralist above some level of detail, but a post-structuralist below it.
Attention Lurkers: Please say hi


I've read nearly everything on less wrong but except for a couple months last summer, I generally don't comment because a) I feel I don't have time, b) my perfectionist standards make me anxious about meeting and maintaining the high standards of discussion here and c) very often someone has either already said what I would have wanted to say or I anticipate from experience that someone will very soon.

3Strange712yEven if you know someone else is going to say it soon, do so yourself and you'll still get some of the credit.
That Magical Click

There's the consequentialist/utilitarian click, and the intelligence explosion click, and the life-is-good/death-is-bad click, and the cryonics click.

I can find a number of blog posts from you clearly laying out the arguments in favor of each of those clicks except the consequentialism/utilitarianism one.

What do you mean by "consequentialism" and "utilitarianism" and why do you think they are not just right but obviously right?

1Document12yIt's linked from the wiki page, but just to shorten the distance: The "Intuitions" Behind "Utilitarianism" [] was what came to mind for me.
3AngryParsley12yHmm... you probably want to read Circular Altruism []. There are different forms of utilitarian consequentialism. The LW wiki has a blurb with links to other useful posts [].
Open Thread: December 2009

d) should be changed to the sparseness of intelligent aliens and limits to how fast even a superintelligence can extend its sphere of influence.

A Less Wrong singularity article?

Interesting, what about either of the following:

A) If X should do A, then it is rational for X to do A.

B) If it is rational for X to do A, then X should do A.

1wedrifid12yFrom what I understand of what Eliezer's position: False False. (If this isn't the case then Eliezer's 'should' is even more annoying than how I now understand it.)
A Less Wrong singularity article?

I'm a moral cognitivist too but I'm becoming quite puzzled as to what truth-conditions you think "should" statements have. Maybe it would help if you said which of these you think are true statements.

1) Eliezer Yudkowsky should not kill babies.

2) Babyeating aliens should not kill babies.

3) Sharks should not kill babies.

4) Volcanoes should not kill babies.

5) Should not kill babies. (sic)

The meaning of "should not" in 2 through 5 are intended to be the same as the common usage of the words in 1.

2Nick_Tarleton12yI don't think there is one common usage of the word "should". (ETA: I asked the nearest three people if "volcanoes shouldn't kill people" is true, false, or neither, assuming that "people shouldn't kill people" is true or false so moral non-realism wasn't an issue. One said true, two said neither.)
0[anonymous]12yI don't think there's one canonical common usage of the word "should". (I'm not sure whether to say that 2-5 are true, or that 2-4 are type errors and 5 is a syntax error.)

Technically, you would need to include a caveat in all of those like, "unless to do so would advance paperclip production" but I assume that's what you meant.

0Eliezer Yudkowsky12yThey all sound true to me.
A Less Wrong singularity article?

I don't think we anticipate different experimental results.

I find that quite surprising to hear. Wouldn't disagreements about meaning generally cash out in some sort of difference in experimental results?

A Less Wrong singularity article?

On your analysis of should, paperclip maximizers should not maximize paperclips. Do you think this is a more useful characterization of 'should' than one in which we should be moral and rational, etc., and paperclip maximizers should maximize paperclips?

3Eliezer Yudkowsky12yA paperclip maximizer will maximize paperclips. I am unable to distinguish any sense in which this is a good thing. Why should I use the word "should" to describe this, when "will" serves exactly as well?
Less Wrong Q&A with Eliezer Yudkowsky: Ask Your Questions

Do you think that morality or rationality recommends placing no intrinsic weight or relevance on either a) backwards-looking considerations (e.g. having made a promise) as opposed to future consequences, or b) essentially indexical considerations (e.g. that I would be doing something wrong)?


Its painfulness.

After some medical procedure, there have been some patients for whom pain is not painful. When asked whether their pain is still there, they will report that the sensation of pain is still there just as it was before, but that they simply don't mind it anymore.

That feature of pain that their pain now lacks is what I am calling its painfulness and that is what is bad about pain.

Joint Distributions and the Slow Spread of Good Ideas

Yes. And since being a maverick has a similar negative expectation for most working people, it seems well-placed to explain the slow spread of good ideas more generally as well.

Outside Analysis and Blind Spots

Great post.

I agree that you identify a very good reason to take care in the use of gender-specific pronouns or anything else that is likely to create in-group, out-group effects.

I also think there probably was a fair amount of attitude polarization on the question of how acceptable it was to make the statement in question.

Sayeth the Girl

Under what conditions do you normally find it necessary to attempt to fully describe a goal?

0Alicorn13yUsually when I'm very, very bored.
Sayeth the Girl

Upvoted because I appreciate Alicorn's efforts and would like to hear additional rational presentations of views in the same neighborhood as her's.

I would bet I also upvoted some of the comments Alicorn is referring to as comments that perpetuate the problem.

Sayeth the Girl

disregard for the autonomy of people =/= thinking of someone in a way that doesn't include respect for his goals, interests, or personhood

I am reading the latter rather literally in much the same way RobinHanson seems to and as I think the author intended.

0SoullessAutomaton13ySorry, I thought it clear I meant some flavor of "all of the above", shortened for readability.
Absolute denial for atheists

Nice. Tying the usage of words to inferences seems to be a generally useful strategy for moving semantic discussions forward.

Absolute denial for atheists

I had negative associations attached to Roko's comment because I started imagining myself with my preferences adopting Roko's suggestions.

This sentence was meant to explain why I was momentarily off-put. I did not mean to imply that I have any ethical problems with the desires mentioned (I don't), though now that you mention it, I wouldn't be too surprised if I do retain some knee-jerk ethical intuitions against them.

Absolute denial for atheists

Have you tried programming in a language with an interactive interpreter, extensive documentation and tutorials, and open source code?

Absolute denial for atheists

I still have very little idea what you mean by 'objectification' and 'objectify people'.

I was momentarily off-put by Roko's comment on the desire to have sex with extremely attractive women that money and status would get. This was because of:

  • the focus on sex, whereas I would desire a relationship.
  • the connotation of 'attractive' which in my mind usually means physical attractiveness, whereas my preferences are dominated by other features of women.
  • the modifier 'extremely' which seems to imply a large difference in utility placed on sex with extremely at
... (read more)
3Alicorn13yI'm sorry you don't understand where I'm coming from. I don't have any bright ideas about how to make it less ambiguous. Is there some reason you are put off when others don't share your desires? If the desire in question was something like "I desire to behave ethically" that would be okay, but there doesn't seem to be anything wrong with wanting sex but no relationship. There are ethical ways to pursue that desire. It's certainly nice that your attraction isn't dominated solely by physical features, but that isn't actually what "attractive" means on a reliable enough basis that I thought it was worth bringing up. Even if "conventionally physically attractive" was what Roko meant, there doesn't seem to be anything obviously wrong with that in light of the focus on sex over a relationship. One person can want to have no-strings-attached sex with multiple conventionally physically attractive women and I can want to settle down in a long-term relationship with a bespectacled dark-haired person with an IQ over 120 and there is no reason to think that these desires can't both be okay simultaneously. I don't see this as any more problematic than the mention of attractiveness in the first place. If it's okay for me to want a spouse with an IQ over 120, presumably it'd be okay for me to want a spouse with an IQ over 140, it'd just make a person satisfying my criteria trickier to find; the same would be true if Roko or anyone else wants to have sex with women several standard deviations above the physical attractiveness mean. Not more than, but "getting a [girl/boy]friend" isn't unloaded language either... (I have been known to use the word "obtain" with respect to a hypothetical future spouse myself, but that's mostly because "marry" would sound redundant.)
The Strangest Thing An AI Could Tell You

"The Fermi paradox is actually quite easily resolvable. There are zillions of aliens teeming all around us. They're just so technologically advanced that they have no trouble at all hiding all evidence of their existence from us."

9[anonymous]12yWho would find that implausible? (Not to say that I can't think of anyone who would find that implausible.)
The Strangest Thing An AI Could Tell You

I thought Chalmers is an analytic functionalist about cognition and only reserves his brand of dualism for qualia.

Can self-help be bad for you?

Yes, but not all self-help needs to involve positive affirmations.

I was going to ask whether repeating positive statements about oneself has actually been recommended on lesswrong. Then I remembered this post. Perhaps that post would have made a more suitable target than the claim that rationalists should win.

Wouldn't a rationalist looking to win simply welcome this study along with any other evidence about what does or does not work?

Open Thread: July 2009

Unless that changes then, I wouldn't particularly recommend programming as a job. I quite like my programming job but that's because I like programming and I don't work in a dilbert cartoon.

Rationality Quotes - July 2009

According to an old story, a lord of ancient China once asked his physician, a member of a family of healers, which of them was the most skilled in the art.

The physician, whose reputation was such that his name became synonymous with medical science in China, replied, "My eldest brother sees the spirit of sickness and removes it before it takes shape and so his name does not get out of the house."

"My elder brother cures sickness when it is still extremely minute, so his name does not get out of the neighborhood."

"As for me, I punc

... (read more)
Open Thread: July 2009

Do you program for fun?

Rationality Quotes - July 2009

Take the thoughts of such an one, used for many years to one tract, out of that narrow compass he has been all his life confined to, you will find him no more capable of reasoning than almost a perfect natural. Some one or two rules on which their conclusions immediately depend you will find in most men have governed all their thoughts; these, true or false, have been the maxims they have been guided by. Take these from them, and they are perfectly at a loss, their compass and polestar then are gone and their understanding is perfectly at a nonplus; and t

... (read more)
Rationality Quotes - July 2009

There is a mathematical style in which proofs are presented as strings of unmotivated tricks that miraculously do the job, but we found greater intellectual satisfaction in showing how each next step in the argument, if not actually forced, is at least something sweetly reasonable to try. Another reason for avoiding [pulling] rabbits [out of the magicians's hat] as much as possible was that we did not want to teach proofs, we wanted to teach proof design. Eventually, expelling rabbits became another joy of my professional life.

-- Edsger Dijkstra

Edit: Added context to "rabbits" in brackets.

Rationality Quotes - July 2009

Thanks for the explanations.

Rationality Quotes - July 2009

Testing shows the presence, not the absence of bugs.

-- Edsger Dijkstra

1davidr13yisomorphic to experiments in science, false and correct theories
What's In A Name?

Here's one way this could be explained: Susie realizes that her name could become a cheap and effective marketing tool if she sells seashells at the seashore. Since that's something she enjoys doing anyway, she does so.

If that's how things are, I wouldn't really call this a cognitive bias.

Cascio in The Atlantic, more on cognitive enhancement as existential risk mitigation

That's a good point, but it would be more relevant if this were a policy proposal rather than an epistemic probe.

2steven046113yI don't see why this being an epistemic probe makes risk per near future time unit more relevant than total risk integrated over time. The whole thing is kind of academic, because for any realistic policy there'd be specific groups who'd be made smarter than others, and risk effects depend on what those groups are.
Cascio in The Atlantic, more on cognitive enhancement as existential risk mitigation

To answer your second question: No, there aren't any historical examples I am thinking of. Do you find many historical examples of existential risks?

Edit: Global nuclear warfare and biological weapons would be the best candidates I can think of.

1HughRistik13yCould you answer my first question, too? Which are the intelligent, well-intentioned, and relatively rational humans you are thinking of? Scientists developing nanotech, biotech, and AI? Policy-makers? Who? How would an example disaster scenario unfold in your view? Are you saying that the very development of nanotech, biotech, and AI would create an elevated level of existential risk? If so, I would agree. A common counter-argument I've heard is that whether we like it or not, someone is going to make progress in at least one of those areas, and that we should try to be the first movers rather than someone less scrupulous. In terms of safety, using AI as an example: World with no AI > World where relatively scrupulous people develop an AI > World where unscrupulous people develop an AI Think about how the world would be if Russia or Germany had developed nukes before the US. Intelligence did allow the development of nukes. Yet given that we already have them, global intelligence would probably decrease the risk of them being used. Let's assume, for the sake of argument, that the mere development of future nanotech, biotech, and AI doesn't go horribly wrong [] and create an existential disaster. If so, then the existential risk will lie in how these technologies are used. I will suggest that there is a certain threshold of intelligence greater than ours where everyone is smart enough not to do globally harmful stunts with nuclear weapons, biotech, nanotech, and AI and/or smart enough to create safeguards where small amounts of intelligent crazy people can't do so either. The trick will be getting to that level of intelligence without mishap.
Cascio in The Atlantic, more on cognitive enhancement as existential risk mitigation

If you decreased the intelligence of everyone to 100 IQ points or lower, I think overall quality of life would decrease but that it would also drastically decrease existential risks.

Edit: On second thought, now that I think about nuclear and biological weapons, I might want to take that back while pointing out that these large threats were predominantly created by quite intelligent, well-intentioned and rational people.

8steven046113yIf you decreased the intelligence of everyone to 100 IQ points or lower, that would probably eliminate all hope for a permanent escape from existential risk. Risk in this scenario might be lower per time unit in the near future, but total risk over all time would approach 100%.
Cascio in The Atlantic, more on cognitive enhancement as existential risk mitigation

You seem to be assuming that the relation between IQ and risk must be monotonic.

I think existential risk mitigation is better pursued by helping the most intelligent and rational efforts than by trying to raise the average intelligence or rationality.

4Roko13yThis claim is false - The reversal test does not require the function risk(IQ) to be monotonic. It only requires that the function is locally monotonic around the current IQ value of 100.
Cascio in The Atlantic, more on cognitive enhancement as existential risk mitigation

And I will suggest in turn that you are guilty of the catchy fallacy name fallacy. The giant cheesecake fallacy was originally introduced as applying to those who anthropomorphize minds in general, often slipping from capability to motivation because a given motivation is common in humans.

I'm talking about a certain class of humans and not suggesting that they are actually motivated to bring about bad effects. Rather all it takes is for there to be problems where it is significantly easier to mess things up than to get it right.

-1Vladimir_Nesov13yI agree, this doesn't fall clearly under the original concept of giant cheesecake fallacy, but it points to a good non-specious generalization of that concept, for which I gave a self-contained explanation in my comment. Aside from that, your reply addresses issues irrelevant to my critique of your assertion. It sounds like a soldier-argument [].
Cascio in The Atlantic, more on cognitive enhancement as existential risk mitigation

I think many of the most pressing existential risks (e.g. nanotech, biotech and AI accidents) come from the likely actions of moderately intelligent, well-intentioned, and rational humans (compared to the very low baseline). If that is right then increasing the number of such people will increase rather than decrease risk.

4HughRistik13yCould you elaborate a bit more on why you think this? Are there any historical examples you are thinking of?
5Roko13yAnd also, this argument is vulnerable to the reversal test. If you think that higher IQ increases existential risk, then you think that lower IQ decreases it. Presumably you don't believe that putting lead in the water supply would decrease existential risks?
3Vladimir_Nesov13yThat's a kind of the giant cheesecake fallacy. Capability increases risk caused by some people, but it also increases the power of other people to mitigate the risks. Knowing about the increase in the capability of these factors doesn't help you in deciding which of them wins.
Intelligence enhancement as existential risk mitigation

Increases in rationality can sometimes lead with some regularity to decreasing knowledge or utility (hopefully only temporarily and in limited domains).

Intelligence enhancement as existential risk mitigation

I suspect you aren't sufficiently taking into account the magnitude of people's irrationality and the non-monotonicity of rationality's rewards. I agree that intelligence enhancement would have greater overall effects than rationality enhancement, but rationality's effects will be more careful and targeted -- and therefore more likely to work as existential risk mitigation.

0MichaelBishop13yCould you elaborate on the shape of the rewards to rationality?
4Roko13yI agree that a world where everyone had good critical thinking skills would be much safer. But getting there is super-tough. Learning is something most people HATE. Rationality - especially stuff involving probability theory, logic, statistics and some basic evolutionary science - requires IQ 100 as a basic prerequisite in my estimation. I will discuss the ways we could get to a rational world, but this post is about merely a more intelligent world.
Intelligence enhancement as existential risk mitigation

I'm not sure intelligence enhancement alone is sufficient. It'd be better to first do rationality enhancement and then intelligence enhancement. Of course that's also much harder to implement but who said it would be easy?

It sounds like you think intelligence enhancement would result in rationality enhancement. I'm inclined to agree that there is a modest correlation but doubt that it's enough to warrant your conclusion.

0Roko13yAll things considered, it seems that giving rationality training is much less likely to work than just telling people that if they take a pill it will make them smarter (and therefore richer).
Rationality Quotes - June 2009

"One can measure the importance of a scientific work by the number of earlier publications rendered superfluous by it."

-- David Hilbert

Rationality Quotes - June 2009

"Muad'Dib learned rapidly because his first training was in how to learn. And the first lesson of all was the basic trust that he could learn. It's shocking to find how many people do not believe they can learn, and how many more believe learning to be difficult. Muad'Dib knew that every experience carries its lesson."

-- Frank Herbert, Dune

Rationality Quotes - June 2009

"Science is what we understand well enough to explain to a computer. Art is everything else we do. ... Science advances whenever an Art becomes a Science. And the state of the Art advances too because people always leap into new territory once they have understood more about the old."

-- Donald Knuth

9gwern13yMarvin Minsky, "Why Programming Is a Good Medium for Expressing Poorly-Understood and Sloppily-Formulated Ideas" []
4komponisto13yRelated: -- Milton Babbitt (from "Contemporary Music Composition and Music Theory as Contemporary Intellectual History", 1972)
Rationality Quotes - June 2009

I like some of the imagery but I wouldn't say whatever the outcome is, it is by definition good.

To continue with the analogy, sometimes our inner book of morals really says one thing while a momentary upset prevents what is written in that book from successfully governing.

Why safety is not safe

You seem to think an FAI researcher is someone who does not engage in any AGI research. That would certainly be a rather foolish researcher.

Perhaps you are being fooled by the fact that a decent FAI researcher would tend not to publicly announce any advancements in AGI research.

-3asciilifeform13yScience as priestcraft: a historic dead end, the Pythagoreans and the few genuine finds of the alchemists nonwithstanding. I am astounded by the arrogance of people who consider themselves worthy of membership in such a secret club, believing themselves more qualified than "the rabble" to decide the fate of all mankind.
0[anonymous]13yWhat do you expect the loser of the bet to repay me with? Ammo?
The Terrible, Horrible, No Good, Very Bad Truth About Morality and What To Do About It

Voted Down. Sorry, Roko.

I don't find Greene's arguments to be valuable or convincing. I won't defend those claims here but merely point out that this post makes it extremely inconvenient to do so properly.

I would prefer concise reconstructions of important arguments over a link to a 377 page document and some lengthy quotes, many of which simply presuppose that certain important conclusions have already been established elsewhere in the dissertation.

As an exercise for the reader demonstrating my complaint, consider what it would take to work out whether Jo... (read more)

0TheAncientGeek5y"This analysis of morality" ,is badly in need of a summary itself, as Hanson pointed out. My summary would be that it is a version of the naturalistic fallacy, the idea that morality is whatever people think it is.
0Roko13yIf you have any more suggestions as to what you're after, do let me know.
0Roko13yI will attempt to do some condensing work at some point today given the comments that we have seen here. Thanks.
Mate selection for the men here

Thanks for clarifying what factors you think are relevant. I agree that those have not been tested.

Mate selection for the men here

The correlations with independent ratings of attractiveness were still .44 and .39. Compared to .04 and -.06 for intelligence, that still supports the conclusion that "sheer physical attractiveness appears to be the overriding determinant of liking."

They also used various personality measures assessing such things as social skills, maturity, masculinity/femininity, introversion/extroversion and self-acceptance. They found predominantly negative correlations (from -.18 to -.08) and only two comparatively small positive correlations .14 and .03.

The correlations with independent ratings of attractiveness were still .44 and .39. Compared to .04 and -.06 for intelligence, that still supports the conclusion that "sheer physical attractiveness appears to be the overriding determinant of liking."

Speaking from a hypothetical PUA's point of view, there are still some uncontrolled-for factors:

  • Degree of a male's control over his rating of physical attractiveness (via choice of clothes, grooming, posture, voice tone, etc.)

  • Male's ability to display desirable characteristics through attitude,

... (read more)
Load More