With log utility, the models are remarkably unconcerned with existential risk, suggesting that large consumption gains that A.I. might deliver can be worth gambles that involve a 1-in-3 chance of extinction

I mean, math is nice and all, but this remarkably feels Not How Humans Actually Function. Not to mention that an assumption of AI benefitting a generic humans uniformly seems laughably naïve since unless you really had some kind of all-loving egalitarian FAI, odds are AI benefits would go to some humans more than others.

This paper seems to me like it belongs to the class of arguments that don't actually use math to find out any new conclusions, but rather just to formalize a point that is so vague and qualitative, it could be made just as well in words. There's no real added value from putting numbers into it since so many important coefficients are unknown or straight up represent opinion (for example, whether existential risk shouldn't carry an additional negative utility penalty beyond just everyone's utility going to zero). I don't feel like this will persuade anyone or shift anything in the discussion.

[-]Richard_Kennaway3y63

In fact, the verbal tl;dr seems to be that if you don't care much about enormously wonderful or terrible futures (because of bounded or sufficiently concave utility functions) then you won't pay much to achieve or avoid them.

[-]gjm3y31

For those who like me didn't find it immediately obvious what the various parameters in "Table 2" mean:

is the economic growth rate per unit time produced by AI, versus $g_{0}$ without it. (You can take "per unit time" to mean "per year"; I don't think anything in the paper depends on what the unit of time actually is, but the without-AI figures are appropriate for a unit of one year.)
$m_{a i}$ is the personal mortality risk per year with AI, versus $m_{0}$ without it. (Simple model where everyone independently either dies or doesn't during each year, with fixed probabilities.)
$γ$ is a risk-aversion parameter: d(utility)/d(wealth) is inversely proportional to the $γ$ power of wealth. So $γ = 1$ is the usual "log utility"; larger values mean you value extra-large amounts of wealth less, enough so that any $γ > 1$ means that the amount of utility you can get from being wealthier is bounded. (Aside: It's not clear to me that "risk aversion" is a great term for this, since there are reasons to think that actual human risk aversion is not fully explained by decreasing marginal utility.)
The values displayed in the table are "existential risk cutoffs" meaning that you favour developing AI that will get you those benefits in economic growth and mortality rate up to the point at which the probability that it kills everyone equals the existential risk cutoff. (It's assumed that any other negative consequences of AI are already factored in to those m and g figures.)

The smallest "existential risk cutoff" in the table is about 2%, and to get it you need to assume that the AI doesn't help with human mortality at all, that it leads to economic growth of 10% per year (compared with a baseline of 2%), and that your marginal utility of wealth drops off really quickly. With larger benefits from AI or more value assigned to enormous future wealth, you get markedly larger figures (i.e., with the assumptions made in the paper you should be more willing to tolerate a greater risk that we all die).

I suspect that actually other assumptions made in the paper diverge enough from actual real-world expectations that we should take these figures with a grain of salt.

[-]Nathan Helm-Burger3y20

I don't think this paper is valuable, because the premises seem absurd. You can't anchor on currently existing humans and current correlations between wealth in our current economy in order to predict the value of sapient beings after a singularity. Do digital humans count? Can you assume that wealth will have decreasing returns to self-modifying rapidly self-duplicating digital humans in the same way it does to meat humans? What about the massive expanse of the future light cone, and whether that gets utilized or neglected due to extinction?

This paper doesn't touch on the true arguments against extinction, and thus massively overvalues options that risk extinction.

[-]gjm3y20

Unless I hallucinated it, an earlier version of this post included a table showing how the "existential risk cutoff" (~= Scott Aaronson's "Faust parameter") varies with (1) risk aversion, (2) anticipated economic growth due to superhuman AI, and (3) anticipated reduction in mortality due to superhuman AI. I'm curious why it was removed.