Finally the true face of the LessWrong admins is revealed. I would do the sensible thing and go back to using myspace.

That's a great idea, I would definitely love to participate!

That is exactly the problem I am trying to address. On one hand, I can't figure out how to estimate the likelihood of a situation. On the other hand, it's quite evident that some people would fake a picture as mentioned above since many people find it of some importance. I just can't figure out how to try and evaluate the likelihood of one versus the other. When should I be confused?

But how can I apply this sort of logic to the problems I've described above? It still seems to me like I need in theory to sum over all of the probabilities in some set A that contains all these improbable events but I just don't understand how to even properly define A, as its boundaries seem fuzzy and various thing "kinda fit" or "doesn't quite fit, but maybe?" instead of plain true and false.

My main point I think, is that this is a more general problem. Some configurations of observations can seem extremely unlikely, yet the sum over all of these configurations might be fairly probable. Like if an airplane has an engine failure above your home town, and is about to crash unto it. The probability of it crashing right near your house is small (if you live in a big town), but it has to crash near someone's house. And that person that had the airplane crash right on his front lawn would go and say "What do you know, what are the odds? So unlikely!".

So while the above example is simple to explain, what happens if someone say's one day as a joke "I hope an airplane won't crash on my house" and on that day an airplane does? That's on its own seems rare enough that it shouldn't be happening every day, or even every year, or maybe even it never happened in history (assuming reasonable stuff like 'not in a time of war' etc ...). But that may happen to someone at some point, and we won't go up and say "that's insane, that couldn't possible be true", because we understand in some level that the probability of observing something with a low probability is very different from the probability of observing specifically that low probability event. And so, maybe that wouldn't happen with an airplane but with a lightning strike, someone saying "I hope lightning won't strike me today" and get struck, or a meteor, or any other of huge number of other situations. So how do we when something doesn't fit the model? Where should I say "I should be confused by this, this phenomenon is ought not to be possible."?

BTW - I am sure that the one in a million events happen all the time is in Methods of Rationality, but it may have some earlier references.

I think it's worth dividing blackmail into two distinct types:

1. Blackmailing on information that is harmful to society.

2. Blackmailing on information that is not harmful to society, but which the victim feels private about.

Your arguments stand somewhat well for the first type. For example, if one is stealing money from the cash register where he works on a weekly basis, then we would not want such behavior to persist. But for the latter type, for example, if someone is secretly a homosexual and is afraid of what his family would say or do if they knew, I don't think we'd like to force him 'out of the closet'.

A possibly more serious problem would be how the extortionist can escalate the stakes (similar to Zvi's argument if I understood it correctly), where one may start with blackmailing the victim about being a homosexual, and proceed to force him to steal money from the cash register in order to have even more leverage on him. In other words, an intelligent blackmailer could potentially start from type 2 but cause type 1 actions to be performed.

Lastly - Blackmailers do no reveal said information to society, making it all better. They would actually rather to never reveal that information (thus losing their ability to blackmail the victim). They instead make personal profit and gain from it which may also allow the victim to persist with his harmful / illicit behavior. In other words, the amount the victim pays is not a simple function of how much his behavior is harmful to society, but depends on how good the blackmailer is and how much he knows. In this regard, it may be worth while to simply tell the authorities (assuming some ideal authorities, yeah I know - not very realistic). In which case they have the means to investigate the matter in depth and enforce the socially accepted punishment for such an offense. Do note that this also means that the victim would not be punished for type (2) blackmails.

So my bottom line is - perhaps giving people incentive to tell the authorities about someone else's illicit behavior is a better way of doing things, assuming the authorities aren't too awful.

Sounds really cool, too bad the 'more details document' is all in Russian. I suppose it's not like I would go to Russia just for an RPG, but it sounds like fun and I would love hearing more details about it.

I think the title is a little bit misleading, and perhaps he didn't put much emphasis on this, but it seems he isn't claiming correct models are generally bad, just that there are also possible downsides to holding correct models and it's probably a good idea to be aware to these flaws when applying these models to reality.

Also, it seems to me as he is defining 'correct model' as a model in which the reasoning is sound and can be used for some applications, however does not necessarily fully describe every aspect of the problem.

What word do you mean? Friendly AI? It's a term (I'm hardly an expert, but I guess wikipedia should be okay for that https://en.wikipedia.org/wiki/Friendlyartificial intelligence )