I know in your hypothetical there are no other effects of the button, but I think a very relevant consideration is how the safety work done now on current models will provide lessons for safety work on future models. I’d assume this would outweigh the warning shot effect. I think warning shots are valuable, but I’m pressing the button until my fingers are blistered.
Surely, other things equal, your best estimate for future voting is current voting. It's more likely that another 20 will upvote than another 20 downvote. If you're only concerned with the outcome, your best strategy will be to downvote. Of course, you may feel really bad if you downvoted a comment below what you think it deserves, because you were responsible.
I didn't read EY as making that point. The 'hit points' analogy suggests that he's giving money priority over other things. Am I wrong?
In any case, there are some things that don't seem measurable in dollar terms: how much money would I accept to drop X% in social status or Y fewer friends? I have no idea. I'm not suggesting that people don't trade these things off, just that money is not a neutral currency with which we can compare the value of any two things. It allows for a imperfect but useful comparison of the subjective value of things for which a ... (read more)
Just noticed that taw makes the same signaling point below.
In our society, this common currency of expected utilons is called "money". It is the measure of how much society cares about something.
What is this 'society' of which you speak?
And why should we expect a common currency of utilions? Money can buy you lots of things, but not anything. Plenty of people give up lucrative careers for more satisfying ones. That doesn't negate your overall argument, but I think you're wrong on that particular point.
My view is that charity has less to do with actually helping the needy than with signaling compass... (read more)
I think a lot of religious signaling works because folks can signal commitment to a particular religious community. While publicly being an atheist/rationalist may make it harder to join a religious group, thus keeping you an atheist, I doubt it commits you to a particular organization. It seems to me that part of the reason religious communities are so stable is that so much of an individual's identity is tied to believing in this particular organization, having these particular goals, or following this particular charismatic leader. Strong and unqualified loyalty to a particular group seems at odds with rationalism.
A persuasive school of thought in the economics of religion suggests that in order to build community, churches often artificially increase barriers to exit and require all sorts of crazy behaviour to signal commitment, thus preventing free-riding. Irrational belief and the accompanying ritual seems to be pretty good at this. I'm not too sure how a rationalist community would fare in this respect...
This is very interesting. One thing stands out to me: Gemini realises that producing synthetic scores is cheating, but then does it anyway after saying “My code is an extension of my judgement.” This seems like a case of searching for a justification which would resolve conflicting priorities and allow for task completion, even if the justification is very weak. If the process is something like "consider various reasons for and against an action until you find a set of reasons all pointing in the same direction," that is worrying. There were two wolves inside Gemini here, but only one of them allowed for easy task completion. In RL environments, this is probably the wolf that gets fed.