As with human aesthetic sense, human morality may be approximations and versions of more absolutely definable optimal solutions to information-theoretic, game-theoretic, social, economic, intelligence, signaling, cooperation problems. Therefore it may be likely that an alien race could share some of the same values as we do, because it may turn out that they are "good" solutions for intelligent culture bearing species in general. But there is nothing in the universe in it self that says that theese optimal solutions, or any value what-so-ever can... (read more)

And to answer Obert's objection that Subhan's position doesn't quite add up to normality: before we knew game theory, evolutionary psychology, and memetics, nothing screened off our moral perceptions/intuitions from a hypothesized objective moral reality, so that was perhaps the best explanation available, given what we knew back then. And since that was most of human history, it's no surprise that morality-as-given feels like normality. But given what we know today, does it still make sense to insist that our meta-theory of morality add up to that normality?

I will try to express some of my points more accurately... A human value, may it concern, knowledge, morality or beauty gets it's meaning from it's emotional base although it may be a frequent value in the space of possible intelligent species. Only minds can attribute value to something. The thing it attributes it to, may be universal or specific, but the thing itself is can not be valued by something other than a mind. To value something is a cognitive, emotional process, not some intrinsic property of some phenomenon. But to believe this as the mind you... (read more)

I think that we need a much better explanation of this word "mind". Supposedly mind space contains a -1 for every 1, but that simply sounds like system space. I honestly think that the ontology has to go deeper here before progress is possible. Similar problem to born postulates and why we aren't Boltzman Brains.

Similar problem to born postulates and why we aren't Boltzman Brains.

Not that this has anything to do with the topic, which everyone is very carefully skating around without addressing: what are operational definitions for right and wrong? When Obert says "Because it seems blue, just as murder seems wrong.", what collection of properties does wrong refer to? For that matter, what does blue ... (read more)

I don't think you have to postulate Space Cannibals in order to imagine rational creatures who don't think murder is wrong. For a recent example, consider Rwanda 1994.

And I think it's quite possible that there might exist moral facts which humans are incapable of perceiving. We aren't just universal Turing machines, after all. Billions of years of evolution might produce creatures with moral blind spots, anologous to the blind spot in the human eye. Just as the squid's eye has no blind spot, a different evolutionary path might produce creatures with a greater or lesser innate capacity to perceive goodness than ourselves.

Maybe this will make it easier:

Obert says "just as murder seems wrong". There is a redundancy in that phrase. What is the redundancy, and why doesn't Obert perceive it as one?

What is the difference between saying something is a rube and not a blegg, and saying that someone appears to be a rube and not a blegg?

What is the difference between saying something is imperceivable, and saying something appears to be imperceivable?

The notion of morality as subjectively objective computation seems a lot closer to Subhan's position than Obert's.

Yes, EY's past positions about Morality are closer to Subhan's than Obert's. But AGI is software programming and hardware engineering, not being a judge or whoever writes laws. I wouldn't suggest deifying EY if your goal is to learn ethics.

But we already know why murder seems wrong to us. It's completely explained by a combination of game theory, evolutionary psychology, and memetics. These explanations screen off our apparent moral perceptions from any other influence. In order words, conditioned on these explanations being true, our moral perceptions are independent of (i.e. uncorrelated with) any possible morality-as-given, even if it were to exist.

Let's try the argument with mathematics: we know why we think 5 is a prime number. It's completely explained by our evolution, experiences, an... (read more)

"But AGI is [...] not being a judge or whoever writes laws."

If Eliezer turns out to be right about the power of recursive self-improvement, then I wouldn't be so sure.

Richard, we can understand how there would be evolutionary pressure to produce an ability to see light, even if imperfect. But what possible pressure could produce an ability to see morality?

"You're not escaping that easily! How does a universe in which murder is wrong, differ from a universe in which murder is right? How can you detect the difference experimentally? If the answer to that is 'No'...

Minor quibble - 'no' is not a sensical answer to any of those questions. Possibly remove the word 'how' from one of them?

Once again, no revelations that I haven't come across on my own, but crystallised and clarified brilliantly. Looking forward to the next few.

It seems to me that Obert makes a faulty interpretation of "there is no reason to talk about a 'morality' distinct from what people want.", but i would like to know what the author thinks. In my view, that assertion says not that ALL MORAL CLAIMS ARE WHIMS, but instead that to understand and parse and compare moral claims we have to resort to wants. In other words, that WANTS ARE THE OBJECT OF MORALITY, THOUGH NOT IT'S MATTER. To understand any moral claim we have to consider how it imparts onto what real, concrete persons feel and desire.

"I want pie" and "I deserve pie" are different, but i don't see how Subhan's arguments aspire to make them equal.

Obert's arguments seem much closer to "how it feels from the inside", Subhan in general does seem to have stronger actual arguments, however:

"For every kind of stone tablet that you might imagine anywhere, in the trends of the universe or in the structure of logic, you are still left with the question: 'And why obey this morality?'" This, to me, smells of zombieism. "for any configuration of matter/energy/whatever, we can ask 'and why should we believe that this is actually conscious rather than just a structure immitating a consciousness?'"

(ZMDavis wrote:) "But AGI is [...] not being a judge or whoever writes laws."

If Eliezer turns out to be right about the power of recursive self-improvement, then I wouldn't be so sure."

Argh. I didn't mean that as a critique on EY's prowess as an AGI theorist or programmer. I doubt Jesus would've wanted people to deify him, just to be nice to eachother. I doubt EY meant for his learning of philosophy to be interpreted as some sort of Moral code, he was just arrogant enough not to state he was sometimes using his list to as a tool to develo... (read more)

So here's a question Eliezer: is Subhan's argument for moral skepticism just a concealed argument for universal skepticism? After all, there are possible minds that do math differently, that do logic differently, that evaluate evidence differently, that observe sense-data differently...

Either Subhan can distinguish his argument from an argument for universal skepticism, or I say that it's refuted by reductio, since universal skepticism fails to the complete impossibility of asserting it consistently + things like moorean facts.

Phillip, you're the one who brought up "deification," in response to my one-line comment, which you seem to have read a lot into. My second comment was intended to be humorous. I apologize for the extent to which I contributed to this misunderstanding.

Eliezer seems to suggest that the only possible choices are morality-as-preference or morality-as-given, e.g. with reasoning like this:

[...] the morality-as-preference viewpoint is a lot easier to shoehorn into a universe of quarks. But I still think the morality-as-given viewpoint has the advantage [...]

But really, evolutionary psychology, plus some kind of social contract for group mutual gain, seems to account for the vast bulk of what people consider to be "moral" actions, as well as the conflict between private individual desires vs. a... (read more)

I think it's probably useful to taboo the word "should" for this discussion. I think when people say you "should" do X rather than Y it means something like "experience indicates X is more likely to lead to a good outcome than Y". People tend to have rule-based rather than consequence based moral systems because the full consequences of one's actions are unforeseeable. A rule like "one shouldn't lie" comes about because experience has shown that lying often has negative consequences for the speaker and listener and p... (read more)

Paul Gowder,

Yes, there are possible minds that do math/logic/deduction differently. Most of these logically possible minds perform even worse than humans in these aspects, and would die out.

In this universe, if one wishes to reach ones goals, one has to choose to (try to) do math/logic/deduction in the correct way; the way that delivers results. What works is determined by the laws of physics and logic that in our universe seem quite coherent and understandable (to a degree, at least).

There's no reason to be skeptical about whether I actually have some goa... (read more)

Subhan's question here, "How does a universe in which murder is wrong, differ from a universe in which murder is right? How can you detect the difference experimentally?" is such a gem.

I wonder if Eliezer intended it as parody.

If somebody said to me "morality is just what we do." If they presented evidence that the whole apparatus of their moral philosophy was a coherent description of some subset of human psychology and sociology. Then that would be enough for me. It's just a description of a physical system. Human morality would be what human animals do. Moral responsibility wouldn't be problematic; moral responsibility could be as physical as gravity if it were psychologically and sociologically real. "I have a moral responsibility" would be akin to "... (read more)

I've thought about Space Cannibals and the like before (i.e. creatures that kill one of the sexes during sexual reproduction). My suspicion is that even if such creatures evolved and survived, by the time they had a civilization, many would be saying to one another, "There really should be a better way..."

Evidence for this is the fact that even now, there are many human beings claiming it is wrong to kill other animals, despite the fact that humans evolved to kill and eat other animals. Likewise, in the ancestral environment, various tribes usual... (read more)

Robin, As Eliezer has pointed out, evolution is a nonhuman optimizer which is in many ways more powerful than the human mind. On the assumption that humans have a moral sense, I don't think we should expect to be able to understand why. That might simply be a problem which is too difficult for people to solve. That aside, a man's virtues benefit the society he lives in; his inclination to punish sin will encourage others to act virtuously as well. If his society is a small tribe of his relatives, then even the weaker forms of kin selection theory can explain the benefit of knowledge of good and evil.

Treating those who do not deserve respect with respect is basically spitting on those who do deserve it, especially those who work hard for it. I think you need to treat those you don't know with the "presumption of respect"; that is, if you don't know that they don't deserve it, assume they do. Borrowed from Smith's "presumption of rationality"; when you argue with someone, assume that they are rational until they demonstrate otherwise.

Richard, would you accept the same argument about God, that we know there is a God but don't really understand how we know, but gosh darn it we feel like there must be one so there must be one? Yes we evolved to help kin, and we expect many but hardly all other species to do this as well. But unless we know whether that behavior is moral we don't know if that is a process that makes our moral intuitions correlate with moral truth.

Richard, we can understand how there would be evolutionary pressure to produce an ability to see light, even if imperfect. But what possible pressure could produce an ability to see morality?

Let's detail the explanation for light to see if we can find a parallel explanation for morality. Brief explanation for light: light bounces off things in the environment in a way which can in principle be used to draw correct inferences about distant objects in the environment. Eventually, some animals evolve a mechanism for doing just this.

Let's attempt the same for ... (read more)

Robin,

Our moral intuitions correspond with moral truths for much the same reason that our rational predictions correspond with more concrete physical truths. A man who ignores reason will stick his hand back in the fire after being burned the first time. Such behavior will kill him, probably sooner rather than later. An man who is blind to good and evil may do quite well for himself, but a society whose citizens ignore virtue will suffer approximately the same fate as the twice-burned fool.

Richard, I agree that some social norms help a society prosper while others can "burn" it. And we have the intuition that morally right acts correspond to social norms that help societies prosper. But we would have had that intuition even if morally right acts had corresponded to the opposite. What evolutionary pressure could have produced the correct intuitions about this meta question?

Constant, I would say that objective illness is just as problematic as objective morality; it's just less obviously problematic because in everyday contexts, we're more used to dealing with disputes about morality than about illness. You mention that "if we select an ill partner for producing offspring we may produce no offspring," and in an evolutionary context, probably we could give some fitness-based account of illness. However, this evolutionary concept of "illness" cannot be the ordinary meaning of the word, because no one actual... (read more)

Constant wrote: So one place where one could critique your argument is in the bit that goes: "conditioned on X being the case, then our beliefs are independent of Y". The critique is that X may in fact be a consequence of Y, in which case X is itself not independent of Y.

Good point, my argument did leave that possibility open. But, it seems pretty obvious, at least to me, that game theory, evolutionary psychology, and memetics are not contingent on anything except mathematics and the environment that we happened to evolve in.

So if I were to draw ... (read more)

I wonder if Eliezer intended it as parody.

Eliezer: You have perhaps already considered this, but I think it would be helpful to learn some lessons from E-Prime when discussing this topic. E-Prime is a subset of English that bans most varieties of the verb "to be".

I find sentences like "murder is wrong" particularly underspecified and confusing. Just what, exactly, is meant by "is", and "wrong"? It seems like agreeing on a definition for "murder" is the easy part.

It seems the ultimate confusion here is that we are talking about instrumental values (... (read more)

My earlier comment is not to imply that I think "maximization of human happiness" is the most preferred goal.

An easily obvious one, yes. But faulty; "human" is a severely underspecified term.

In fact, I think that putting in place a One True Global Goal would require ultimate knowledge about the nature of being, to which we do not have access currently.

Possibly, the best we can do is come up with plausible global goal that suits us for medium run, while we try to find out more.

That is, after all, what we have always done as human beings.

Wanting to murder doesn't make it right. Nothing makes anything morally right.

Robin,

I don't understand your counterfactual.

"Good" and "Evil" are the names for what people perceive with their moral sense. I think we've agreed that this perception correlates to something universally observable (namely, social survival), so these labels are firmly anchored in the physical world. It looks to me like you're trying to assign these names to something else altogether (namely, something which does not correlate with human moral intuitions), and it's not clear to me how this makes sense.

Richard, if morality just meant social norms that help societies prosper, then of course we have little problem understanding how the two could be correlated, and how we could come to know about them. But if morality means something else, then we face the much harder question of how it is we could know about this something else.

For those impatient to know where Eliezer is going with this series, it looks like he gaves us a sneak preview a little more than a year ago. The answer is morality-as-computation.

Eliezer, hope I didn't upset your plans by giving out the ending too early. When you do get to morality-as-computation, can you please explain what exactly is being computed by morality? You already told us what the outputs look like: "Killing is wrong" and "Flowers are beautiful", but what are the inputs?

EY: "human cognitive psychology has not had time to change evolutionarily over that period"

Under selective pressures, human populations can and have significantly changed in less than two thousand years. Various behavioral traits are highly heritable. Genghis Khan spread his behavioral genotype throughout Asia. (For this discussion this is a nitpick but I dislike seeing false memes spread.)

re: FAI and morality

From my perspective morality is a collection of rules that make cooperative behavior beneficial. There are some rules that should apply to ... (read more)

Robin,

I don't know how people are capable of discerning moral truths. I also don't know how people are capable of discerning scientific or mathematical truths. It seems to me that these are similar capabilities, and the one is no more suprising or unlikely than the other.

Richard, while there are surely many details we would like to understand better, surely we understand the basic outline of how we discern scientific and mathematical truths. For example, in math we use contradiction to eliminate possible implications of axiom sets, and in science we use empirical results to eliminate possible abstract theories. We have nothing remotely similar in morals. You never said whether you approved of a similar argument about knowledge of God.

Z. M. Davis writes: ... objective illness is just as problematic as objective morality

I would argue that to answer Robin's challenge is not necessarily to assert that there is such a thing as objective illness.

Accounts have been given of the pressure producing the ability to see beauty (google sexual selection or see e.g. this). This does not require that there is some eternal beauty written in the fabric of the universe - it may be, for example, that each species has evolved its own standard of beauty, and that selection is operating on both sides, i.e., ... (read more)

"what possible pressure could produce an ability to see morality?"

Unlike the other Richard, I don't think we "see" morality with a special "sense", or anything like that. But if we instead understand morality as a rational idealization, building on our perfectly ordinary general capacity for systematizing judgments so as to increase their overall coherence (treating like cases alike, etc.), then there's no great mystery here.

Dynamically Linked writes: But, it seems pretty obvious, at least to me, that game theory, evolutionary psychology, and memetics are not contingent on anything except mathematics and the environment that we happened to evolve in.

According to Tegmark "there is only mathematics; that is all that exists". Suppose he is right. Then moral truths, if there are any, are (along with all other truths) mathematical truths. Unless you presuppose that moral truths cannot be mathematical truths then you have not ruled out moral truths when you say that so-and... (read more)

Robin:

Discarding false mathematical and scientific conjectures is indeed much easier than discarding false moral conjectures. However, as Eliezer pointed out in an earlier post, a scientist who can come up with a hypothesis that has a 10% chance of being true has already gone most of the way from ignorance to knowledge. I would argue that hypothesis generation is a poorly-understood nonrational process in all three cases. A mathematician who believes he has found truth can undertake the further steps of writing a formal proof and submitting his work to ... (read more)

Constant, if moral truths were mathematical truths, then ethics would be a branch of mathematics. There would be axiomatic formalizations of morality that do not fall apart when we try to explore their logical consequences. There would be mathematicians proving theorems about morality. We don't see any of this.

Isn't it simpler to suppose that morality was a hypothesis people used to explain their moral perceptions (such as "murder seems wrong") before we knew the real explanations, but now we find it hard to give up the word due to a kind of memetic inertia?

Constant, if moral truths were mathematical truths, then ethics would be a branch of mathematics. There would be axiomatic formalizations of morality that do not fall apart when we try to explore their logical consequences. There would be mathematicians proving theorems about morality. We don't see any of this.

If Tegmark is correct, then everything is mathematics. Do you dispute Tegmark's claim that "there is only mathematics; that is all that exists"? Do you think your argument is any good against Tegmark's hypothesis? Will you tell Tegmark, &qu... (read more)

What's so bad about morality being a mere human's construct - in other words, the notion that there is no "stone tablet" of morals? In fact, I think the notion that morality exists objectively, like some looming Platonic figure, raises more questions than would be solved by such a condition.

I think the best way to construct this "morality" is just to say that it's got a quasi-mathematical existence, it's axiomic, and it's all augmented by empirical/logical reasoning.

Why accept it, why be moral? I feel the same way about this question as I do about the question of why somebody who believes "if A, then B," and also believes that A, should also believe that B.

sigh

Not even close, any of you ;)

The question of whether there are any moral givens or not is analagous to the question of whether there are any mathematical givens (which was covered by E.Yudkowsky in an earlier series). Unfortunately, he doesn't seem to have learned the lesson.

Firstly, in the series on mathematics it was (correctly I think) put forward that even mathematics is not in fact, a priori or axiomatic. 2+2=4 for instance, can be empirically based on the observation that when you have two apples, and you add another two apples, you end up with... (read more)

Geddes, if you can't keep the condescension out of your comments - just present the raw arguments, if you have any - then I'll have to ban you here, too. Just FYI. Also, your comments should be shorter.

@ Ian C. Couldn't Subhan claim that as a restatement of his own position? His notion of wanting clearly encompasses more than mere whims. Perhaps he would say that a certain subset of desires, objectively grounded in the constitution of the mind, count as moral impulses.

Actually, is Subhan meant to be male? Apologies if not.

@Lake - I think Subhan is only about whims. Yes, he sees that values are tied closely to human nature, but only uses that to argue against Obert. What Obert should have pointed out is that he goes from "there is not one true morality" to "there is only preference" without arguing why those are the only two possibilities.

FYI, it's physics that is fundamental. Math is deeper than our theories of physics - it's deeper than all our theories, because it makes up the languages we use to create and express them - but physics itself is deeper than everything.

Similarly, if there are any moral givens, I would agree that they have to result in real empirical differences.

Again, we could never percieve moral givens directly (since they are abstract)

I gestured at one possible answer to that question. A situation has a moral dimension if it engages moral emotions - which can presumably be listed.

Still doesn't tell us what 'moral' means. We've just changed the category we stick that label on. What defines 'moral emotions'? Is it an arbitrary grouping, or do we use the label to refer to certain properties that things in that grouping possess? People seem to use the term in the latter way - so what properties are they?

Basic scientific methodology - you can't study what you can't produce a provisional definition for. Once you have that, you can learn more about what's defined, but you don't get anywhere without that starting point.

Basic scientific methodology - you can't study what you can't produce a provisional definition for. Once you have that, you can learn more about what's defined, but you don't get anywhere without that starting point.

The first concepts that more less denoted, say, water, may have included things which today we would reject as not water (e.g., possibly clear alcohol), failed to distinguish water from things dissolved in the water, and excluded forms of water (such as steam and ice). The very first definitions of water were probably ostensive definitions (thi... (read more)

Are you willing to accept an ostensive and potentially erroneous definition of morality that may very well be subject to revision as knowledge improves?

Are you willing to have a neverending discussion, with everyone talking past each other, and no working definition for the central concept we're supposed to be examining?

Are you willing to have a neverending discussion, with everyone talking past each other, and no working definition for the central concept we're supposed to be examining?

I'm not in charge of the discussion, so it's not a question of what I'm willing to do. I've told you how to get the starting definition you're looking for. As I said: you can start with an ostensive definition by listing examples of evil acts. Then you can find common elements. For example, it might become apparent, after surveying them, that evil acts have in common that they all have vic... (read more)

It seems the ultimate confusion here is that we are talking about instrumental values . . . before agreeing on terminal values . . .

If we could agree on some well-defined goal, e.g. maximization of human happiness, we could much more easily theorize on whether a particular case of murder would benefit or harm that goal.

denis bider, I would not be surprised to learn that refraining from murder is a terminal value for Eliezer. Eliezer's writings imply that he has hundreds of terminal values: he cannot even enumerate them all.

Defn. "Murder" is killing under particular circumstances, e.g., not by uniformed soldiers during a war, not in self-defense, not by accident.

Great dialog, which I think can be summarized in Nietzsche's aphorism: "Morality is the herd-instinct in the individual."

Actually, I think the dialog could have been a lot shorter if it became clear earlier on that preference (as in morality-as-preference) referred not to individual preference but the "preference of the collective". Which is to say, morality is determined by evolutionary psychology. There are however two assumptions built into the evolutionary psychological explanation of morality which ought to be made explicit.

The fir... (read more)

My comment is not charitable enough towards the CEVists. I ask the moderator to delete it, I will now submit a replacement.

It seems the ultimate confusion here is that we are talking about instrumental values . . . before agreeing on terminal values . . .

If we could agree on some well-defined goal, e.g. maximization of human happiness, we could much more easily theorize on whether a particular case of murder would benefit or harm that goal.

denis bider, under the CEV plan for singularity, no human has to give an unambiguous definition or enumeration of his or her terminal values before the launch of the seed of the superintelligence. Consequently, those who lean toward th... (read more)

As I said: you can start with an ostensive definition by listing examples of evil acts. Then you can find common elements.

There aren't necessarily any common elements, besides utterly trivial ones. If you look at examples of misspelled words in various languages and examine their individual properties, you won't find what unites them in a category. You have to understand their relationship to the spelling rules in the various languages - rules which themselves are likely to be incompatible and mutually incoherent - to understand what properties make t... (read more)

Quick comment: Terren Suydam's version of "evolutionary psychology" is not the academically accepted one. Conventional academic evolutionary explanations of morality rely on neither group selection nor selection on cultures.

Richard, once we can see how to eliminate math or science views, then it doesn't seem particularly puzzling that people can generate plausible views to consider. The obvious hypothesis is that they generate many views in their mind, apply crude but effective filters, and then only tell others about the few best ones. So of course the views they propose will fare far better than average when tested against consistency and data.

Quick comment: Terren Suydam's version of "evolutionary psychology" is not the academically accepted one. Conventional academic evolutionary explanations of morality rely on neither group selection nor selection on cultures.

Be that as it may, I would have to say that an explanation of morality made strictly in terms of the academically accepted version of evolutionary psychology is not possible. I'm not trying to redefine the term - just saying what else would be necessary to make an explanation of morality on that basis possible.

There aren't necessarily any common elements, besides utterly trivial ones.

Maybe, maybe not. You won't know without looking. You have to start somewhere.

If you look at examples of misspelled words in various languages and examine their individual properties, you won't find what unites them in a category.

But then, what about correctly spelled words? There will be many observable systematic relationships between those. I happen to think you have the analogy backwards. In the good/evil dichotomy, it is the evil acts, not the not-evil acts, which are narrowly ... (read more)

Terren Suydam: "The first is that one has to adopt the group-selection stance."

(Technical jargon nitpick.)

In studying evolutionary biology, "group-selection" has a specific meaning, an individual sacrifices its own fitness in order to improve the group fitness. I.e., individual loss for a group gain. E.g., suppose you have a species that consists of many small family groups. Suppose a mutation produces a self-sacrificing individual in one of the groups. His fitness is slightly lower but his family group fitness is higher. His group tend... (read more)

Nobody told Galileo and Newton what the rules generating the world's behavior were, but they were able to go a long way toward figuring them out.

Nobody gets anywhere, most especially in philosophy, without rigorous definitions of the relevant concepts. You rely on vague, intuitive understandings, and you accomplish nothing.

In studying evolutionary biology, "group-selection" has a specific meaning, an individual sacrifices its own fitness in order to improve the group fitness.

I think it's quite limiting to think strictly in terms of genetics, because there is more than one level of description going on when it comes to selection pressure.

It is interesting to take that step back and view the culture as an individual. The human super-organism (e.g., a tribe, or more generally, a culture) competes with others for resources. It consumes, metabolizes, and excretes, which... (read more)

Terren, I would doubt that changes between cultures are best explained by an evolutionary process--cf. "No Evolutions for Corporations or Nanodevices." There may be a selection effect in that cultures with guns are more likely to persist, but that's different from saying that selection pressures play a really important role in designing the particular features of a culture. So I am given to understand.

There may be a selection effect in that cultures with guns are more likely to persist, but that's different from saying that selection pressures play a really important role in designing the particular features of a culture.

That's what I'm saying - selection pressures are important in determining cultural features, because those features in turn determine a culture's viability. The global-level organization of a culture - including its moral code, political organization, and other important social structures - are key considerations in what makes a culture... (read more)

Terren Suydam: "So genetics is not the whole story, and that's what I mean by group selection."

I use the term "multilevel selection" for what you are describing. I agree it has been important.

E.g., there has been selection between different species. Species with genomes that supported rapid adaptation to changing environments and that supported quick diversification when expanding into new niches spread far and wide. (Beetles have been extremely successful with around 350,000 known species.) Other specie branches died out. The genetic m... (read more)

>Geddes, if you can't keep the condescension out of your comments - just present the raw arguments, if you have any - then I'll have to ban you here, too. Just FYI. Also, your comments should be shorter.

But all I'm asking for is an explanation as to why decision theory works? Perhaps someone like R.Hanson could explain?

After all, I know (admittedly only in very general terms) what the explanation for thermodynamics is (the underlying explanation is in the concepts of mechanics- energy, force etc.). Also, I know (in general), why probability theory wor... (read more)

« Then you believe in universally compelling arguments processed by a ghost in the machine. For every possible mind whose utility function assigns terminal value +1, mind design space contains an equal and opposite mind whose utility function assigns terminal value -1. »

That's true but that doesn't say anything about the sustainability of that given mind design (the possibility for the mind design to survive, either by having the individual to survive, or to create new individuals with similar mind designs).

A mind design that would value its own death ... (read more)

It would be a pitiful mind indeed that demanded authoritative answers so strongly, that it would forsake all good things to have some authority beyond itself to follow.

Having an authority to follow might actually be that mind's one good thing. Maybe it really likes having such authority beyond itself.

While humans obviously don't consider that their only good this and there is human variation (I don't think everyone values it, I'm only certain at least a few do), it seems pretty clear to me that one of our good things, as in stuff we find worth bothering... (read more)

By changing a mind, you can change what it prefers; you can even change what it believes to be right; but you cannot change what is right.  Anything you talk about, that can be changed in this way, is not 'right-ness'.

If the characters were real people, I'd say here Obert is "right" while having a wrong justification. Just extrapolate the evolutionary origins of moral intuitions into any society in approximate technological stasis. "Rightness" is how the evolutionarily stable strategy feels like from the inside, an... (read more)

Obert:  "Why doesn't their society fall apart in an orgy of mutual killing?"

Subhan:  "That doesn't matter for our purposes of theoretical metaethical investigation.  But since you ask, we'll suppose that the Space Cannibals have a strong sense of honor—they won't kill someone they promise not to kill; they have a very strong idea that violating an oath is wrong.  Their society holds together on that basis, and on the basis of vengeance contracts with private assassination companies.  But so far as the actual killing is concerned, t