As far as I can tell, this possibility of an exponentially-paced intelligence explosion is the main argument for folks devoting time to worrying about super-intelligent AI now, even though current technology doesn't give us anything even close. So in the rest of this post, I want to push a little bit on the claim that the feedback loop induced by a self-improving AI would lead to exponential growth, and see what assumptions underlie it.

I think few AI safety advocates believe this. It's much more common to expect growth to be faster than exponential. As you point out, exponential growth is a knife-edge phenomenon.

As far as I can tell, very few people actually think that intelligence growth would exhibit an actual mathematical singularity

This is actually a pretty common view---not a literal singularity, but rapid technological acceleration until natural resource limitations (e.g. on total available solar energy and raw minerals) start binding. If you look at the history of technological progress, it looks a whole lot more like a hyperbola than like an exponential curve, so the hyperbolic growth forecast isn't so insane. It's the person arguing that growth rates are going to stop at 3% who is arguing against the bulk of historical precedent (and whose predecessors would have been wrong if they'd expected growth to stop at 0.3% or 0.03% or 0.003%...).

this seems instead to be a metaphor for exponential growth.

I think "singularity" usually either follows Vinge's use (as the point beyond which you can't predict what will happen, because the future is guided by actors smarter than you are) or as a reference to the dynamic that would produce a mathematical singularity if left unchecked.

In a more typical endogenous growth model, output is the product of physical capital (e.g. how many computers you have) and a technology factor (e.g. how smart you are). You can either invest in producing more capital (building more computers) or doing research (becoming smarter). On these models, even returns of $x^{ϵ}$ still lead to a mathematical singularity (while constant technology leads to exponential growth).

From this perspective, you are investigating whether there is an intelligence explosion with finite capital. If productivity grows sublinearly with inputs, you need to build more machines (and ultimately extract more resources from nature) in order to grow really fast. This might suggest that getting to a singularity would take years rather than weeks, but doesn't much change the qualitative conclusion or substantially change the urgency (especially given that the early phase of takeoff would be driven by moving resources over from lower productivity areas into higher productivity areas).

I think it's a mistake to think of "productivity is linear in effort" as the "no diminishing returns" model, and to consider it a degenerate extreme case. Linear returns is the model where doubling inputs leads to doubled outputs. A priori, it's nearly as natural for constant additional effort leads to doubling of efficiency, so we need to actually look at the data to distinguish.

(It seems more theoretically natural---and more common in practice---for each clever trick to lead to a 10% increase in efficiency, then for each clever trick to lead to an absolute increase of 1 unit of efficiency.)

In semiconductors, as you point out, output has increased exponentially over time. Research investment has also increased exponentially, but with a significantly smaller exponent. So on your model the curve appears to be $x^{\alpha }$ for $\alpha >1$.

The performance curves database contains many interesting time series, and you'll note that the y-axis is typically exponential. They don't track inputs, so it's a bit hard to draw conclusions, but comparing to overall increases in R&D investment it looks like superlinear returns are probably quite common.

A few years ago Katja looked into the rate of algorithmic progress, and found that it was very approximately comparable to the rate of progress in hardware (though it's hard to know how much of that comes from realizing increasing economies of scale w.r.t. compute), across a range of domains. Algorithms seem like a particularly relevant domain to the current discussion.

Hi all,

Thanks for the very thoughtful comments; lots to chew on. As I hope was clear, I'm just an interested outside observer, and have not spent very long thinking about these issues, and don't know much of the literature. (My blog post ended up as a cross post here because I posted it to facebook, and asked if anyone could point me to more serious literature thinking about this problem, and a commenter suggested that I should crosspost here for feedback)

I agree that linear feedback is more plausible if we think of research breakthroughs as producing multiplicative gains, a simple point that I hadn't thought about.

Eliezer did exactly this calculation in an old LW post. Unfortunately I have no idea how to find it. Fortunately the calculation comes out the same no matter who does it!

(1) As Paul noted, the question of the exponent alpha is just the question of diminishing returns vs returns-to-scale.

Especially if you believe that the rate $f=f\left(R\right)$ is a product of multiple terms (like e.g. Paul's suggestion $f=R^{{\alpha }_t}\cdot R^{{\alpha }_a}$ with one exponent for computer tech advances and another for algorithmic advances) then you get returns-to-scale type dynamics (over certain regimes, i.e. until all fruit are picked) with finite-time blow-up.

(2) Also, an imho crucial aspect is the separation of time-scales between human-driven research and computation done by machines (transistors are faster than neurons and buying more hardware scales better than training a new person up to the bleeding edge of research, especially considering Scott's amusing parable of the alchemists).

Let's add a little flourish to your model: You had the rate of research $I$ and the cumulative research $R$ ; let's give a name $C$ to the capability of the AI system. Then, we can model ${\partial }_{t}R=I=f\left(R\right)=g\left(C\right)=g\left(h\right(R\left)\right)$ . This is your model, just splitting terms into $h$, which tells us how hard AI progress is, and $g$ which tells us how good we are at producing research.

Now denote by $q=q\left(C\right)$ the fraction of work that absolutely has to be done by humans, and by $\epsilon$ the speed-up factor for silicon over biology. Amdahl's law gives you $g\left(C\right)=\frac{1}{q\left(C\right)+\epsilon (1-q(C\left)\right)C}$ , or somewhat simplified $g\left(C\right)\ge \frac{1}{q+\epsilon C}$ . This predicts a rate of progress that first looks like $1/q$ , as long as human researcher input is the limiting factor, then becomes $1/\left(\epsilon C\right)$ when we have AIs designing AIs (recursive self-improvement, aka explosion), and then probably saturates at something (when the AI approaches optimality).

The crucial argument for fast take-off (as far as I understood it) is that we can expect $q\left(C\right)$ to hit $q=0$ at some cross-over $C^{\ast }$, and we can expect this to happen with a nonzero derivative ${\partial }_{C}q({}^{}$