Makes sense. But on this question too I'm confused—has some evidence in the last 8 years updated you about the old takeoff speed debates? Or are you referring to claims Eliezer made about pre-takeoff rates of progress? From what I recall, the takeoff debates were mostly focused on the rate of progress we'd see given AI much more advanced than anything we have. For example, Paul Christiano operationalized slow takeoff like so:
Given that we have yet to see any such doublings, nor even any discernable impact on world GDP:
... it seems to me that takeoff (in this sense, at least) has not yet started, and hence that we have not yet had much chance to observe evidence that it will be slow?
I think one of the (many) reasons people have historically tended to miscommunicate/talk past each other so much about AI timelines, is that the perceived suddenness of growth rates depends heavily on your choice of time span. (As Eliezer puts it, "Any process is continuous if you zoom in close enough.")
It sounds to me like you guys (Thane and Ryan) agree about the growth rate of the training process, but are assessing its perceived suddenness/continuousness relative to different time spans?
I have thought of that "Village Idiot and Einstein" claim as the most obvious example of a way that Eliezer and co were super wrong about how AI would go, and they've AFAIK totally failed to publicly reckon with it as it's become increasingly obvious that they were wrong over the last eight years
I'm confused—what evidence do you mean? As I understood it, the point of the village idiot/Einstein post was that the size of the relative differences in intelligence we were familiar with—e.g., between humans, or between humans and other organisms—tells us little about the absolute size possible in principle. Has some recent evidence updated you about that, or did you interpret the post as making a different point?
(To be clear I also feel confused by Eliezer's tweet, for the same reason).
It seems to me that at least while I worked there (2017-2021), CFAR did try to hash this out properly many times, we just largely failed to converge. I think we had a bunch of employees/workshop staff over the years who were in fact aiming largely or even primarily to raise the sanity waterline, just in various/often-idiosyncratic ways.
we tended in our public statements/fundraisers to try to avoid alienating all those hopes, as opposed to the higher-integrity / more honorable approach of trying to come to a coherent view of which priorities we prioritized how much and trying to help people not have unrealistic hopes in us, and not have inaccurate views of our priorities
I... wish to somewhat-defensively note, fwiw, that I do not believe this well-describes my own attempts to publicly communicate on behalf of CFAR. Speaking on behalf of orgs is difficult, and I make no claim to have fully succeeded at avoiding the cognitive bases/self-serving errors/etc. such things incentivize. But I certainly earnestly tried, to what I think was (even locally) an unusual degree, to avoid such dishonesty.
(I feel broadly skeptical the rest of the org's communication is well-described in these terms either, including nearly all of yours Anna, but ofc I can speak most strongly about my own mind/behavior).
1I basically think you're being unfair here, so want to challenge you to actually name these or retract.
... So that's my response to the charge that the param estimates are overly complicated. But I want to respond to one other point you make
It sounds like we're talking past each other, if you think I'm making two different points. The concern I'm trying to express is that this takeoff model—by which I mean the overall model/argument/forecast presented in the paper, not just the literal code—strikes me as containing confusingly much detail/statistics/elaboration/formality, given (what seems to me like) the extreme sparsity of evidence for its component estimates.
the paper is still very clear and explicit about its limitations
I grant and (genuinely) appreciate that the paper includes many caveats. I think that helps a bunch, and indeed helps on exactly the dimension of my objection. In contrast, I think it probably anti-helped to describe the paper as forecasting "exactly how big" the intelligence explosion will be, in a sense constrained by years of research on the question.
It seems to me that demand for knowledge about how advanced AI will go, and about what we might do to make it go better, currently far outstrips supply. There are a lot of people who would like very much to have less uncertainty about takeoff dynamics, some of whom I expect might even make importantly different decisions as a result.
... and realistically, I think many of those people probably won't spend hours carefully reading the report, as I did. And I expect the average such person is likely to greatly overestimate the amount of evidence the paper actually contains for its headline takeoff forecast.
Most obviously, from my perspective, I expect most casual readers to assume that a forecast billed as modeling "exactly how big" the intelligence explosion might be, is likely to contain evidence about the magnitude of the explosion! But I see no evidence—not even informal argument—in the paper about the limits that determine this magnitude, and unless I misunderstand your comments it seems you agree?
If AIs that are about as good as humans at broad skills (e.g. software engineering, ML research, computer security, all remote jobs) exist for several years before AIs that are wildly superhuman
I'm curious how likely you think this is, and also whether you have a favorite writeup arguing that it's plausible? I'd be interested to read it.
I agree the function and parameters themselves are simple, but the process by which you estimate their values is not. Your paper explaining this process and the resulting forecast is 40 pages, and features a Monte Carlo simulation, the Cobb-Douglas model of software progress, the Jones economic growth model (which the paper describes as a “semi-endogenous law of motion for AI software”), and many similarly technical arcana.
To be clear, my worry is less that the model includes too many ad hoc free parameters, such that it seems overfit, than that the level of complexity and seeming-rigor is quite disproportionate to the solidity of its epistemic justification.
For example, the section we discussed above (estimating the “gap from human learning to effective limits”) describes a few ways ideal learning might outperform human learning—e.g., that ideal systems might have more and better data, update more efficiently, benefit from communicating with other super-smart systems, etc. And indeed I agree these seem like some of the ways learning algorithms might be improved.
But I feel confused by the estimates of room for improvement given these factors. For example, the paper suggests better “data quality” could improve learning efficiency by “at least 3x and plausibly 300x.” But why not three thousand, or three million, or any other physically-possible number? Does some consideration described in the paper rule these out, or even give reason to suspect they’re less likely than your estimate?
I feel similarly confused by the estimate of overall room for improvement in learning efficiency. If I understand correctly, the paper suggests this limit—the maximum improvement in learning efficiency a recursively self-improving superintelligence could gain, beyond the efficiency of human brains—is "4-10 OOMs," which it describes as equivalent to 4-10 "years of AI progress, at the rate of progress seen in recent years."
Perhaps I’m missing something, and again I'm sorry if so, but after reading the paper carefully twice I don’t see any arguments that justify this choice of range. Why do you expect the limit of learning efficiency for a recursively self-improving superintelligence is 4-10 recent-progress-years above humans?
Most other estimates in the paper seem to me like they were made from a similar epistemic state. For example, half the inputs to the estimate of takeoff slope from automating AI R&D come from asking 5 lab employees to guess; I don't see any justification for the estimate of diminishing returns to parallel labor, etc. And so I feel worried overall that readers will mistake the formality of the presentation of these estimates as evidence that they meaningfully constrain or provide evidence for the paper’s takeoff forecast.
I realize it is difficult to predict the future, especially in respects so dissimilar from anything that has occurred before. And I think it can be useful to share even crude estimates, when that is all we have, so long as that crudeness is clearly stressed and kept in mind. But from my perspective, this paper—which you describe as evaluating “exactly how dramatic the software intelligence explosion will be”!—really quite under-stresses this.
This model strikes me as far more detailed than its inputs are known, which worries me. Maybe I’m being unfair here, I acknowledge the possibility I’m misunderstanding your methodology or aim—I’m sorry if so!—but I currently feel confused about how almost any of these input parameters were chosen or estimated.
Take your estimate of room for “fundamental improvements in the brain’s learning algorithm,” for example—you grant it's hard to know, but nonetheless estimate it as around “3-30x.” How was this range chosen? Why not 300x, or 3 million? From what I understand the known physical limits—e.g., Landauer's bound, the Carnot limit—barely constrain this estimate at all. I'm curious if you disagree, or if not, what constrains your estimate?
I don't think they are quite different. Christiano's argument was largely about the societal impact, i.e. that transformative AI would arrive in an already-pretty-transformed world:
I claim the world is clearly not yet pretty-transformed, in this sense. So insofar as you think takeoff has already begun, or expect short (e.g. AI 2027-ish) timelines—I personally expect neither, to be clear—I do think this takeoff is centrally of the sort Christiano would call "fast."