There's a form of honesty and coordination that I've found many do not understand or appreciate: openly stating what a hypothetical evil version of you would do in a situation. For instance, when I attempt to team up with someone in an adversarial game, I explain the most inconvenient case where our goals differ, but still assuming that I'm myself sane, attempting to maximize probability of winning.
A notable upside is that it combines quite well with honesty. Especially in deception-based board games, where good players hold any inconsistency against you. If you're not trying to claim anything and neutrally enumerate the options aloud, that's only neutral. As long as you do that consistently. And this might only work when there's a culture around that. In the real world, such a culture rarely exists.
However, I liberally do this, and unsurprisingly it's not always a good idea. Many people just don't consider hypotheticals, and instead ignore them or forget they're hypotheticals. I'm sure I make that mistake sometimes, too. The second issue is that just bringing up the scenario associates you with it, which is by itself, a problem. A phenomenon almost like The Copenhagen Interpretation of Ethics makes the problem worse; why would a well-meaning person bother about thinking of such scenarios, unless they're just waiting for the right moment to backstab you?
Security requires a particular mindset. Security professionals — at least the good ones — see the world differently. They can’t walk into a store without noticing how they might shoplift. They can’t use a computer without wondering about the security vulnerabilities. They can’t vote without trying to figure out how to vote twice. They just can’t help it.
-The Security Mindset, Schneier on Security
It's been working out decently though, as I can afford to filter both my friends and jobs such that they'll like or at least tolerate this. Misunderstandings unfortunately happen, not infrequently. Also there's questionable upside of cultivating the aura of a Dark Lord, which I happen to find both psychologically secure and amusing. And in general, I like promoting a culture where open discussion of such hypotheticals is possible.
Every now and then, I do security consulting for a living. Not the authoritative "as long as these boxes are checked, we're going to give it our seal of approval" kind, but the one that considers potential threats and the incentives around them. One of the hardest aspects to manage is insider threats, especially corporate espionage. In scenarios where you need to suggest that someone in the upper management would be an adversary, one has to be careful to not step on anyone's toes with hypotheticals. Instead, I've said things like "imagine that someone with access to these sensitive files, for instance me, leaks them to the media". If I don't include the "for instance me", it sounds like an accusation and I've seen people get defensive about that. But with it, I've sometimes gotten comments like "why would you do that", and not in the useful incentive-analysis sense.
I used to complain about certain security measures, and security theater is still a thing I despise having to deal with. But most measures are actually useful for catching incompetent and unlucky people. I'm, ironically and to my slight surprise, too afraid to publicly spell out the standard example you can probably guess, as it might actually cause issues while traveling to countries that read all your online content. Anyway, responsiveness to incentives tends to correlate with capability of making things happen. The barriers should only reach as high as the marginal cost justifies, as described in The optimal amount of fraud is non-zero.
The whole idea has something to do with counter-signaling. Only when it's clear that they're smart enough to not publicly speak of crimes they're planning to commit, such talk of the hypotheticals makes them more trustworthy. Or perhaps it's more about trust; if doing something seems uncharacteristic of you, then it's safer for you to mention it. And conversely, only when I feel secure enough can I afford to be open in this way. I suppose Yudkowsky's Meta-Honesty has similar social dynamics, especially around glomarization.
-HPMoR, chapter 7
There's a form of honesty and coordination that I've found many do not understand or appreciate: openly stating what a hypothetical evil version of you would do in a situation. For instance, when I attempt to team up with someone in an adversarial game, I explain the most inconvenient case where our goals differ, but still assuming that I'm myself sane, attempting to maximize probability of winning.
A notable upside is that it combines quite well with honesty. Especially in deception-based board games, where good players hold any inconsistency against you. If you're not trying to claim anything and neutrally enumerate the options aloud, that's only neutral. As long as you do that consistently. And this might only work when there's a culture around that. In the real world, such a culture rarely exists.
However, I liberally do this, and unsurprisingly it's not always a good idea. Many people just don't consider hypotheticals, and instead ignore them or forget they're hypotheticals. I'm sure I make that mistake sometimes, too. The second issue is that just bringing up the scenario associates you with it, which is by itself, a problem. A phenomenon almost like The Copenhagen Interpretation of Ethics makes the problem worse; why would a well-meaning person bother about thinking of such scenarios, unless they're just waiting for the right moment to backstab you?
It's been working out decently though, as I can afford to filter both my friends and jobs such that they'll like or at least tolerate this. Misunderstandings unfortunately happen, not infrequently. Also there's questionable upside of cultivating the aura of a Dark Lord, which I happen to find both psychologically secure and amusing. And in general, I like promoting a culture where open discussion of such hypotheticals is possible.
Every now and then, I do security consulting for a living. Not the authoritative "as long as these boxes are checked, we're going to give it our seal of approval" kind, but the one that considers potential threats and the incentives around them. One of the hardest aspects to manage is insider threats, especially corporate espionage. In scenarios where you need to suggest that someone in the upper management would be an adversary, one has to be careful to not step on anyone's toes with hypotheticals. Instead, I've said things like "imagine that someone with access to these sensitive files, for instance me, leaks them to the media". If I don't include the "for instance me", it sounds like an accusation and I've seen people get defensive about that. But with it, I've sometimes gotten comments like "why would you do that", and not in the useful incentive-analysis sense.
I used to complain about certain security measures, and security theater is still a thing I despise having to deal with. But most measures are actually useful for catching incompetent and unlucky people. I'm, ironically and to my slight surprise, too afraid to publicly spell out the standard example you can probably guess, as it might actually cause issues while traveling to countries that read all your online content. Anyway, responsiveness to incentives tends to correlate with capability of making things happen. The barriers should only reach as high as the marginal cost justifies, as described in The optimal amount of fraud is non-zero.
The whole idea has something to do with counter-signaling. Only when it's clear that they're smart enough to not publicly speak of crimes they're planning to commit, such talk of the hypotheticals makes them more trustworthy. Or perhaps it's more about trust; if doing something seems uncharacteristic of you, then it's safer for you to mention it. And conversely, only when I feel secure enough can I afford to be open in this way. I suppose Yudkowsky's Meta-Honesty has similar social dynamics, especially around glomarization.