BurntVictory — LessWrong

LESSWRONG
LW

The Kitty Genovese Equation

Someone's in trouble. You can hear them from your apartment, but you can't tell if any of your neighbors are already rushing down, or already calling the police. It's time sensitive, and you've got to decide now: is it worth spending those precious minutes, or not?

Let's define our variables:

Cost to victim of nobody helping: $C$

cost to each bystander of intervening: $k < C$

Number of bystanders: $N >= 2.$ (Since $k < C$ , for $N = 1$ it's always right to intervene.)

Analysis:

Suppose the bystanders all simultaneously decide whether to intervene or not, with probability p. Then expected world-utility is $U_{C, k, N} (p) =$ $- C (1 - p)^{N} - k p N$

Utility is maximized when $0 = d U / d p = N C (1 - p)^{N - 1} - k N$ ; In other words, when $(1 - p)^{N - 1} = \frac{k}{C}$ . Let $α = \frac{k}{C}$ .... (read more)

BurntVictory's Shortform

BurntVictory

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Replying toBuying Value, not Price

BurntVictory7y

Buying Value, not Price

The LessWrongy framework I'm familiar with would say that value = expected utility, so it takes potential downsides into account. You're not risk-averse wrt your VNM utility function, but computing that utility function is hard in practice, and EV calculations can benefit from some consideration of the tail-risks.

Replying toThe Game Theory of Blackmail

BurntVictory7y

The Game Theory of Blackmail

Schelling's The Strategy of Conflict seems very relevant here; a major focus is precommitment as a bargaining tool. See here for an old review by cousin_it.

Iterated chicken seems fine to test, just as a spinoff of the IPD that maps to slightly different situations. (I believe that the iterated game of mutually modeling each other's single-shot strategy is different from iterating the game itself, so I don't think Abram's post necessarily implies that iterated chicken is relevant to ASI blackmail solutions.)

Speaking of iterated games, one natural form of blackmail is for the blackmailee to pay an income stream to the blackmailer; that way, at each time-step they're paying their fair price for the good of [not having their secret revealed between time t and time t+1]. Here's a well-cited paper that discusses this idea in the context of nuclear brinksmanship: Schwarz & Sonin 2007.

Replying toPrivacy

BurntVictory7y*

Privacy

It's true the net effect is low to first order, but you're neglecting second-order effects. If premia are important enough, people will feel compelled to Goodhart proxies used for them until those proxies have less meaning.

Given the linked siderea post, maybe this is not very true for insurance in particular. I agree that wasn't a great example.

Slack-wise, uh, choices are bad. really bad. Keep the sabbath. These are some intuitions I suspect are at play here. I'm not interested in a detailed argument hashing out whether we should believe that these outweigh other factors in practice across whatever range of scenarios, because it seems like it would take a lot of time/effort for me to actually build good models here, and opportunity costs are a thing. I just want to point out that these ideas seem relevant for correctly interpreting Zvi's position.

Replying toPrivacy

BurntVictory7y

Privacy

The post implies it is bad to be judged. I could have misinterpreted why, but that implication is there. If judge just meant "make inferences about" why would it be bad?

As Raemon says, knowing that others are making correct inferences about your behavior means you can't relax. No, idk, watching soap operas, because that's an indicator of being less likely to repay your loans, and your premia go up. There's an ethos of slack, decisionmaking-has-costs, strategizing-has-costs that Zvi's explored in his previous posts, and that's part of how I'm interpreting what he's saying here.

But it also helps in knowing who's exploiting them! Why does it give more advantages to

... (read more)

Replying toPrivacy

BurntVictory7y

Privacy

I found this pretty useful--Zvi's definitely reflecting a particular, pretty negative view of society and strategy here. But I disagree with some of your inferences, and I think you're somewhat exaggerating the level of gloom-and-doom implicit in the post.

>Implication: "judge" means to use information against someone. Linguistic norms related to the word "judgment" are thoroughly corrupt enough that it's worth ceding to these, linguistically, and using "judge" to mean (usually unjustly!) using information against people.

No, this isn't bare repetition. I agree with Raemon that "judge" here means something closer to one of its standard usages, "to make inferences about". Though it also fits with the colloquial "deem unworthy for baring [understandable] flaws",... (read more)

Replying toQuestion: MIRI Corrigbility Agenda

BurntVictory7y

Question: MIRI Corrigbility Agenda

The CHAI reading list is also fairly out of date (last updated april 2017) but has a few more papers, especially if you go to the top and select [3] or [4] so it shows lower-priority ones.

(And in case others haven't seen it, here's the MIRI reading guide for learning agent foundations.)

Replying toAI development incentive gradients are not uniformly terrible

BurntVictory7y

AI development incentive gradients are not uniformly terrible

Oh wait, yeah, this is just an example of the general principle "when you're optimizing for xy, and you have a limited budget with linear costs on x and y, the optimal allocation is to spend equal amounts on both."

Formally, you can show this via Lagrange-multiplier optimization, using the Lagrangian $L (x, y) = x y - λ (a x + b y - M)$ . Setting the partials equal to zero gets you $λ = y / a = x / b$ , and you recover the linear constraint function $a x + b y = M$ . So $a x = b y = M / 2$ . (Alternatively, just optimizing $x \frac{M - a x}{b}$ works, but I like Lagrange multipliers.)

In this case, we want to maximize $p q + (1 - p) r q_{0} = p (q - r q_{0}) - r q_{0}$ , which is equivalent to optimizing $p * (q - r q_{0})$ . Let's define $w$ $=$ $q - r q_{0}$ , so we're optimizing $p * w$ .

Our constraint function is defined by the tradeoff between $p$ and $w$ . $p (k) = (.5 - p_{0}) k + p_{0}$ , so $k = \frac{p - p_{0}}{.5 - p_{0}}$ . $w (k) = (r - 1) q_{0} k + q_{0} - r q_{0} = (r - 1) q_{0} (k - 1)$ , so $k = \frac{- w}{(1 - r) q_{0}} + 1 = \frac{p - p_{0}}{.5 - p_{0}}$ .

Rearranging gives the constraint function $\frac{.5 - p_{0}}{(1 - r) q_{0}} w + p = .5$ . This is indeed linear, with a total 'budget' $M$ of .5 and a p-coefficient $b$ of 1. So by the above theorem we should have $1 * p = .5 / 2 = .25$ .

Replying toAI development incentive gradients are not uniformly terrible

BurntVictory7y*

AI development incentive gradients are not uniformly terrible

I think your solution to "reckless rivals" might be wrong? I think you mistakenly put a multiplier of q instead of a p on the left-hand side of the inequality. (The derivation of the general inequality checks out, though, and I like your point about discontinuous effects of capacity investment when you assume that the opponent plays a known pure strategy.)

I'll use slightly different notation from yours, to avoid overloading p and q. (This ends up not mattering because of linearity, but eh.) Let $p_{0}, q_{0}$ be the initial probabilities for winning and safety|winning. Let $k$ be the capacity variable, and without loss of generality let $k$ start at $0$ and end... (read more)

Replying toDrexler on AI Risk

BurntVictory7y

Drexler on AI Risk

Yeah, I worry that competitive pressure could convince people to push for unsafe systems. Military AI seems like an especially risky case. Military goals are harder to specify than "maximize portfolio value", but there are probably reasonable proxies, and as AI gets more capable and more widely used there's a strong incentive to get ahead of the competition.