AI development incentive gradients are not uniformly terrible

[-]Rohin Shah7y60

Coming back to this, I realize that though I summarized this post in Alignment Newsletter #33, I didn't post my opinion here, so doing that now:

The model in Racing to the precipice had the unintuitive conclusion that if teams have more information (i.e. they know their own or other’s capabilities), then we become less safe, which puzzled me for a while. Their explanation is that with maximal information, the top team takes as much risk as necessary in order to guarantee that they beat the second team, which can be quite a lot of risk if the two teams are close. While this is true, the explanation from this post is more satisfying -- since the model has a discontinuity that rewards taking on risk, anything that removes the discontinuity and makes it more continuous will likely improve the prospects for safety, such as not having full information. I claim that in reality these discontinuities mostly don't exist, since (1) we're uncertain about who will win and (2) we will probably have a multipolar scenario where even if you aren't first-to-market you can still capture a lot of value. This suggests that it likely isn't a problem for teams to have more information about each other on the margin.

That said, these models are still very simplistic, and I mainly try to derive qualitative conclusions from them that my intuition agrees with in hindsight.

[-]rk7y30

I think this is a great summary (EDIT: this should read "I think the summary in the newsletter was great").

That said, these models are still very simplistic, and I mainly try to derive qualitative conclusions from them that my intuition agrees with in hindsight.

Yes, I agree. The best indicator I had of making a mathematical mistake was whether my intuition agreed in hindsight

[-]Rohin Shah7y20

(Fyi, this was only my opinion, the summary is in the newsletter. I usually don't post the summary on the post itself, since it is typically repeating the post in a manner that doesn't generate new insights.)

[-]habryka7y50

I feel like most of the bad incentives for developing unsafe AGI comes from the positive incentive gradient of marginally stronger AI. I.e. you could model it as a repeated game with hundreds of rounds, where every turn you get resources and power proportional to your total investment in capabilities. This seems to model reality much better (i.e. financial incentives are basically just short-term, and you don't get to build a 100 person research team if you don't show anything profitable from it), and also seems like where the real incentive problem stems from.

[-]rk7y10

Yes, that seems an important case to consider.

You might still think the analysis in the post is relevant if there are actors that can shape the incentive gradients you talk about: Google might be able to focus its sub-entities in a particular way while maintaining profit or a government might choose to implement more or less oversight over tech companies.

Even with the above paragraph, it seems like the relative change-over-time in resources and power of the strategic entities would be important to consider, as you point out. In this case, it seems like (known) fast takeoffs might be safer!

[-]rk7y20

Comment here if you have structure/writing complaints for the post

[-]rk7y20

Comment here if you are worried about info-hazard-y-ness of talking about AI races

[-]rk7y20

I talked to a couple of people in relevant organisations about possible info hazards for talking about races (not because this model is sophisticated or non-obvious, but because it contributes to general self-fulfilling chattering). Amongst those I talked to, they were not worried about (a) simple pieces with at least some nuance in general and (b) this post in particular

[-]rk7y20

Comment here if there are maths problems

[-]BurntVictory7y*40

I think your solution to "reckless rivals" might be wrong? I think you mistakenly put a multiplier of q instead of a p on the left-hand side of the inequality. (The derivation of the general inequality checks out, though, and I like your point about discontinuous effects of capacity investment when you assume that the opponent plays a known pure strategy.)

I'll use slightly different notation from yours, to avoid overloading p and q. (This ends up not mattering because of linearity, but eh.) Let $p_{0}, q_{0}$ be the initial probabilities for winning and safety|winning. Let $k$ be the capacity variable, and without loss of generality let $k$ start at $0$ and end at $k_{m}$ . Then $p (k) = \frac{.5 - p_{0}}{k_{m}} k + p_{0},$ and $q (k) = \frac{r q_{0} - q_{0}}{k_{m}} k + q_{0}$ . So $p^{'} = \frac{.5 - p_{0}}{k_{m}}$ , so $\frac{p}{p^{'}} = \frac{p * k_{m}}{.5 - p_{0}}$ . And $q^{'} = \frac{r q_{0} - q_{0}}{k_{m}}$ , so $\frac{- q^{'}}{q} = \frac{q_{0} (1 - r)}{q * k_{m}}$ .

Therefore, the left-hand side of the inequality, $- \frac{p q^{'}}{p^{'} q}$ , equals $\frac{p}{.5 - p_{0}} * \frac{q_{0} (1 - r)}{q}$ . At the initial point $k = 0$ , this simplifies to $\frac{p_{0}}{.5 - p_{0}} (1 - r)$ .

Let's assume $α = 1$ . The relative safety of the other project is $β = \frac{r q_{0}}{q}$ , which at $k = 0$ simplifies to $r$ .

Thus we should commit more to capacity when $1 - r > \frac{p_{0}}{.5 - p_{0}} (1 - r)$ , or $1 > \frac{p_{0}}{.5 - p_{0}}$ , or $.25 > p_{0}$ . This is a little weird, but makes a bit more intuitive sense to me than $q_{0} + p_{0}$ or $q_{0} - p_{0}$ mattering.

[-]rk7y20

Yes, you're quite right!

The intuition becomes a little clearer when I take the following alternative derivation:

Let us look at the change in expected value when I increase my capabilities. From the expected value stemming from worlds where I win, we have $(p * q)^{'} = p^{'} * q + p * q^{'}$ . For the other actor, their probability of winning decreases at a rate that matches my increase in probability of winning. Also, their probability of deploying a safe AI doesn't change. So the change in expected value stemming fro m worlds where they win is $- p^{'} * r * q$ .

We should be indifferent to increasing capabilities when these sum to 0, so $p^{'} * q + p * q^{'} = p^{'} * r * q$ .

Let's choose our units so $k_{m} = 1$ . Then, using the expressions for $q^{'}$ from your comment, we have $r q_{0} p_{0}^{'} = p_{0}^{'} q_{0} + p_{0} q_{0} (r - 1)$ .

Dividing through by $q_{0}$ we get $r p_{0}^{'} = p_{0}^{'} + p_{0} (r - 1)$ . Collecting like terms we have $(r - 1) * p_{0}^{'} = p_{0} * (r - 1)$ and thus $p_{0}^{'} = p_{0}$ . Substituting for $p_{0}^{'}$ we have $\frac{1}{2} - p_{0} = p_{0}$ and thus $p_{0} = \frac{1}{4}$

[-]BurntVictory7y10

Oh wait, yeah, this is just an example of the general principle "when you're optimizing for xy, and you have a limited budget with linear costs on x and y, the optimal allocation is to spend equal amounts on both."

Formally, you can show this via Lagrange-multiplier optimization, using the Lagrangian $L (x, y) = x y - λ (a x + b y - M)$ . Setting the partials equal to zero gets you $λ = y / a = x / b$ , and you recover the linear constraint function $a x + b y = M$ . So $a x = b y = M / 2$ . (Alternatively, just optimizing $x \frac{M - a x}{b}$ works, but I like Lagrange multipliers.)

In this case, we want to maximize $p q + (1 - p) r q_{0} = p (q - r q_{0}) - r q_{0}$ , which is equivalent to optimizing $p * (q - r q_{0})$ . Let's define $w$ $=$ $q - r q_{0}$ , so we're optimizing $p * w$ .

Our constraint function is defined by the tradeoff between $p$ and $w$ . $p (k) = (.5 - p_{0}) k + p_{0}$ , so $k = \frac{p - p_{0}}{.5 - p_{0}}$ . $w (k) = (r - 1) q_{0} k + q_{0} - r q_{0} = (r - 1) q_{0} (k - 1)$ , so $k = \frac{- w}{(1 - r) q_{0}} + 1 = \frac{p - p_{0}}{.5 - p_{0}}$ .

Rearranging gives the constraint function $\frac{.5 - p_{0}}{(1 - r) q_{0}} w + p = .5$ . This is indeed linear, with a total 'budget' $M$ of .5 and a p-coefficient $b$ of 1. So by the above theorem we should have $1 * p = .5 / 2 = .25$ .

LESSWRONG
LW

LESSWRONG
LW

21

AI development incentive gradients are not uniformly terrible

21

21

A partly discontinuous model

Chances & resources

Concrete cases

Against failure

If unfriendliness is much, much worse than your enemy winning

Against a cautious player

Reckless Rivals

Armstrong, Bostrom & Shulman 2013