I disagree. All the scam artist has to know is your method of coming to your conclusions. Once he knows that then he can probably exploit you depending on his cleverness (and then it becomes an arms race). If anything, trying to defend yourself from being manipulated in that way would probably be extremely difficult in of itself. Either way, my initial guess is that your methodology would still be superficial pattern-matching, but it would just be a deeper, more complex level of it.

This seems to be what Eliezer is doing with all the various scenarios. He's testing his methodology against different attacks and different scenarios. I'm just suggesting is to change your viewpoint to the Bad Guy. Rather than talk about your reliable reasoning, talk about the bad guy and how he can exploit your reasoning.

But wouldn't you just be suckered by sufficiently smart and attentive scam artists?

Isn't this more of social recognition of a scam?

While there are decision-theoretic issues with the Original Pascal's Wager, one of the main problems is that it is a scam ("You can't afford not to do it! It's an offer you can't refuse!"). It seems to me that you can construct plenty of arguments like you just did, and many people wouldn't take you up on the offer because they'd recognize it as a scam. Once something has a high chance of being a scam (like taking the form of Pascal's Wager), it won't get much more of your attention until you lower the likelihood that it's a scam. Is that a weird form of Confirmation Bias?

But nonetheless, couldn't the AI just function in the same way as that? I would think it would need to learn how to identify what is a trick and what isn't a trick. I would just try to think of it as a Bad Guy AI who is trying to manipulate the decision making algorithms of the Good Guy AI.

One way to save face in socially painful situations is to try to appear as though you are doing it all on purpose, to present yourself, for example, as a 'free spirit' rather than a 'loser.' I know for sure that I did this a lot as a child, and even now I have a tendency to emphasize the 'weird' things that I do, to look like I do them confidently and deliberately.

I always associated this behavior more with machismo. The idea that "acting like you know what you're doing" is more important than "knowing what you're doing." Certainly in social situations, but especially in sexual situations, I never want to signal doubt in my actions, even if my actions turn out to be silly or stupid (which is hilariously often).

Mathematicians - along with scientists - discover new things (what is a proof other than a discovery of a new mathematical property). That's what their job is. In order for Ethicists to be comparable, wouldn't they need to discover new ethics?

Yea, Konkvistador supplied well.

What would be an example of a "Normative human essential"?

Re-reading Greenberg's article makes me want to compose some classical dubstep.

Just wanted to clarify before I let this go.

I am skeptical about your model of aesthetics. I think the model that allows you to compare so easily cross-genre is not the actual model that you use for your aesthetics. All I'm asking is that you double-check to make sure that the model you use actually fits, and you often are able to make these cross-genre comparisons (not comparing genres but comparing songs within different genres).

It is your comparison that baffles me, not the result of the comparison.

The second of the above quotes is something I have claimed. The first one is a response to something I have not claimed. There is a straw man at play.

Well yes, I was using an incorrect model of you.

Valuing Weird Al over classical does not make me less intelligent. It speaks more about my general aesthetic preference for melding a conceptual meaning in closely with the melody, rhythm, tone, etc. For me the concepts themselves seem to be a part of music to a far greater extent than for most people I have compared myself to. I do not consider this to be a weakness of mine.

I was really just commenting on the drama of the last part of your post. Valuing Weird Al over classical isn't something bizarre to me.

The issue I have is the comparison between the two. Different music is for different things. To just say "Well if I'm gonna listen to music then I'll always pick Weird Al over classical," well, that's not all there is to the aesthetics of music.

Edit: Especially with things like epic film scores. Classical music tends to fit this niche quite well, and I would be surprised if you honestly disagreed with that.