This is a linkpost for https://www.ai-2040.com/
New Comment
Mr Kokotajlo, one of the authors of this AI 2040 scenario, describes it as follows:
If you read the scenario, you'll see that the regulations are mostly about what people can do with giant compute clusters, and not about the ideas themselves. The ideas themselves are required to be totally transparent to the public.
Although regulations on giant compute clusters would give humanity more time, they might not avert extinction by themselves. If AI researchers continue to be able to communicate without restriction, the research community might discover (and I'm tempted to say, "will probably discover") and publish a machine-learning algorithm efficient enough to make an AI superhumanly capable, even when running on modest hardware. I'm not the only one who thinks that. Here is Steve Byrnes writing 12 months ago:
...I’m not sure that actual existing efforts towards delaying AGI are helping. I think the path from here to AGI is bottlenecked by researchers playing with toy models, and publishing stuff on arXiv and GitHub. … Once the new paradigm is known and developed, the actors able to train ASI from scratch will probably number in the tens of thousands, spread all around the world. We’
1
1
1Many would reply that these restrictions on dissemination of knowledge will drastically slow down AI research. Yes: the drastic slowing constitutes most of the benefit of imposing the restrictions.
Plan A largely doesn't agree with this. Since they expect the slowdown treaty to eventually fall apart--at which point we revert to the status quo arms race, which we all agree is bad--they argue for slowing only insofar as it allows alignment research to outpace general AI research.
Scott Alexander had a nice little graphic on this point in his ACX post on Plan A:
Plan A also seems to largely discount the possibility of smaller research labs discovering some new paradigm that is capable of becoming an ASI without massive amounts of compute. I largely agree with this view, since ~90% of algorithmic progress is scale/compute-dependent, but either way this seems like an important crux.
11In my humble opinion, the AI 2040 document falls short of being such a plan because it does nothing to stop or slow research that might find (and I'm tempted to say, "will probably find") a fundamentally more efficient learning algorithm than what the leading labs are currently using.
This is overstated. Plan A involves significant attempts to slow down algorithmic progress. See e.g.:
...In the past, companies have trained bigger and better AIs using both compute scaling (bigger training runs) and software progress (advances in AI algorithms—new paradigms, better training recipes, better data, etc.). Now, the Consortium tries to steer things so that the majority of improvement comes from increasing training compute.
Algorithms are information; it’s inherently difficult to stop them from proliferating, and the total research transparency means we aren’t even trying. [FN: There are a few nuances here. Some algorithmic progress is easily communicated (e.g., new architectures, better optimization algorithms), while other types cannot easily diffuse (e.g., huge libraries of RL environments, hardware-software codesign, scale or compute dependent algorithms). Regulations that the US and China a
Plan A involves significant attempts to slow down algorithmic progress
Plan A does not advocate for what I consider the most potent brake on that "progress", namely restrictions enforced with penalties on dissemination of new knowledge from one researcher to another (and related measures like bans on founding a new AI lab or seeking additional investment for an existing lab).
there’s no longer as much incentive to race to discover new AI paradigms and more powerful algorithms, because companies wouldn’t be able to hoard such discoveries and profit greatly from them.
Most of the advances in AI so far have not been the result of any hoarding, but rather of researchers and labs freely giving away knowledge. The most salient example is Google Deep Mind's freely giving away the knowledge described in the "Attention is all you need" paper that started the transformer revolution on which the success of OpenAI and Anthropic depended, and I got the impression from my brief study of the history of the field that every breakthrough in machine learning on which the transformer breakthrough depended was also freely given away shortly after its discovery. Although I know little about the h...
15
The Chinese and US government will probably be able to build covert projects with more compute and knowledge than outsiders. It seems like you're proposing a regime where, once extremely efficient ASI algorithms are discovered, both the US government and the Chinese government would know about it. ("Chinese officials and AI researchers would be a constant presence in US AI labs, and vice versa".) So then they would both be able to secretly defect on the deal and run many ASIs, unknown to the other party, if they so pleased. How are you hoping that the deal will remain stable beyond this point?
(USG + China being sufficiently convinced about misalignment risk that they each unilaterally refuse to run their ASIs, even knowing that the other party could do it without their knowledge? Not impossible, but a significantly higher bar than how bought into the risk you need to be to have the bilateral deal work up until that point.)
(Or maybe you just accept that many ASIs will be launched at that point, and the main goal of the anti-proliferation stuff was just to buy more time for earlier alignment research and/or to prevent lone wolf misuse once ASI is developed?)
6
I think that if an extremely efficient ASI algorithm is discovered and disseminated to the public, it will almost certainly be too late to prevent our extinction (or some other fate just as bad) unless perhaps the discovery and dissemination is done by someone competent like the leaders of MIRI, Steven Byrnes or John Wentworth. (Note that the latter 2 have never worked for an AI lab.)
4
I was trying to understand: Do you have much more hope about the situation if an extremely efficient ASI algorithm is discovered and not immediately disseminated to the public, but where the Chinese and US government both have access to it and can run it without the others' knowledge; and if so why? That seems like an essential piece of info to understand the cost/benefit of moving from the high-transparency world to the world where USG and China are overseeing each others' tech but aren't publishing.
(My two parantheses were me speculating about possible answers to this question.)
2
For me the main benefit of restrictions on publication and other forms of dissemination of breakthroughs, discoveries, insights and plain old technical information about AI is to delay the creation of an ASI. If the ASI is a very efficient learning algorithm, then it would definitely be better if only Washington and Beijing have it and are effectively preventing its dissemination, but it would still be very bad news: IMHO human extinction would still follow within 3 years with p = .95.
Any delay in the arrival of ASI gives humanity more time for someone to come up with some miracle to get us out of our dire situation. The nature of that miracle I probably cannot guess. If I had to guess, I might guess that space aliens will show up and save us from extinction while imposing some of their values on us, values that we would consider bizarre, or something vaguely like that.
4
It sounds like you are proposing a more hardcore version of Plan A or Plan S, where individual researchers are prohibited from talking to each other about certain kinds of ideas?
I think that would have some benefits, but also some costs. For example, restrictions on speech of the sort you want are going to hurt public epistemics probably, when it comes to assessing AI risks and safety cases. Also, it's generally bad to restrict speech for the usual reasons. Lukas Finnveden's comments elsethread are good. I'm open to doing the more hardcore thing if the political will for it manifests and we can work out a way to mitigate the downsides.
7
Yes. Even better would be to dissolve the labs, make it illegal to form a new lab, to accept investment or donations to do AI research, to pay or fund an AI researcher, to train to become a researcher, to train others in research, etc, but unless and until such a drastic ban becomes feasible, any restrictions on a researcher's ability to publish or to disseminate an insight, breakthrough or discovery would IMHO be helpful although (as your document explains or at least unambiguously implies) for Washington to restrict dissemination from the US to foreign powers has the negative effect of encouraging those foreign powers to invest more in AI research, and it is worth a lot to avoid that.
I infer from your choice of the phrase "assessing AI risks and safety cases" that you prefer and expect the labs to continue to create and deploy models (i.e., the ones assessed by the world to be worth the risk) whereas I prefer a blanket ban on the creation and deployment of new models. Of course, the political will for a blanket ban might never materialize, in which case your argument for protecting public epistemics becomes more persuasive, but still on my models the main reason for hoping for good public epistemics is so that the public starts making loud calls to stop the research.
On my models, merely regulating the research in complicated ways while allowing the research to continue fails to lower extinction risk much. "We the public, the broader AI research community and the governments of the world are going to pay close attention to what the researchers at the major labs are up to, and when we see things we don't like, we're going to apply strong pressure on them," is not much of a plan in my eyes. It does almost nothing to reduce my fear of the labs. But I must admit that I have yet to do more than skim AI 2040. (There is a rhetorical advantage to replying quickly that I could not resist.) Is there a more concrete plan in there than what I just summarized?
(Models that
5
That's reasonable. We put Plan S in there as an available option because we do think it's a serious proposal and might even be better than Plan A.
2
I've been modifying my comment (bad habit) for 30 min after you posted yours, which is not really fair to you. I've stopped now.
AI 2040 has given me things to think about.
4
~~I think this is unlikely given AI scaling laws. Algorithmic improvements could drastically decrease the amount of training required but capabilities could still be limited at a given model size and compute requirement. In other words you could have AI with a human brain's plasticity and it wouldn't matter if it doesn't also have sufficient size.~~
Edit: Never mind, I just noticed that in the AI 2040 scenario, AI progress is supposed to mostly come from compute improvements, with algorithmic improvements deliberately suppressed. So the impact of a low hanging fruit algorithmic breakthrough is much higher than a counterfactual scenario where algorithmic improvements are allowed to continue and global compute rollout is slowed instead.
“I wish it need not have happened in my time,” said Frodo. “So do I,” said Gandalf, “and so do all who live to see such times.”
Is there any reason you haven't included human intelligence enhancement as part of scenario S?
Given the level of political will and international coordination in the story, why can't they just dismantle the compute supply chain?
If I understand correctly, the main argument agains Plan S is that at some point the global pause agreement will break down, and then we will be back where we are right now, and the race restarts again at a break-neck speed.
But what if part of the pause deal is that, both in China and in US allies, we destroy a large chunk of the existing GPUs, destroy the fabs, destroy the cutting-edge EUV machines, destroy the equipment necessary to build the EUV machines and disperse the teams working at all these companies so institutional knowledge is lost?
Once the compute supply chain is dismantled, governments can pay attention that no new cutting-edge chip fabs or necessary equipments are made - something that seems much easier to enforce than the restrictions on dangerous algorithmic progress in Plan A.
My understanding is that this wouldn't have huge effects outside the AI industry. While there would be a huge stock market crash and it would be expensive to compensate the the affected companies, I'm not sure the financial loss would be more than 2x bigger ...
1If they have the political will to do Plan A, they very well might also have the political will to dismantle the compute supply chain. This would be a variant of Plan S.
I think this is plausibly as good or better than Plan A, not sure. One issue with it is that a covert project with, say, 100k GPUs doesn't really confer much geostrategic advantage in Plan A, but in Plan S, it might. Imagine: It's 2040. The economy has recovered from the compute supply chain being dismantled; people have learned to live without computers. But negotiations for how to restart AI progress safely and transparently and in a power-distributed way are dragging on and on and it seems like it's basically never going to reach agreement. Meanwhile, the covert project has managed to make an OOM or two of algorithmic progress since 2030, and is just a few years away from fully automating AI R&D, at which point they'll probably have ASI within a few years of that...
Idk. We have a model of AI progress + model of black sites that tries to model situations like this. I don't think it's obvious either way how it would go.
My other objection is that I feel a lot of despair when thinking in near-mode about the Plan A proposal of slowing down algorithmic progress.
----
India's leading company publishes a paper about a training dataset they used to make the user-experience for their AI smoother. The Indian regulator apparently green-lighted it, but they are known for their lenient approach. Some academics notice that the Indian model is now not just smoother to use, but it subtly feels like it's smarter than other models, even though it doesn't directly show on the standard benchmarks. The American AISI starts getting emails from academics, explaining that they feel like the Indian model got smarter, and that they think the paper the Indians published was too high-level to fully reconstruct what they did.
Meanwhile the American AISI is also getting emails from academics complaining about how the European AI is too biased against minorities, and how the Brazilian AI has too high persuasive capabilities, and how the compute cluster in the ocean is harming the fish. They are also tasked with defending the US companies from the totally unfair complaints coming from China and India that they are not transpare...
8
I think I'm just not as afraid of covert projects. In the world that has enough political will for Plan A or compute supply dismantling, I think it's likely that neither the US nor China even wants to attempt a covert project, because they are both genuinely afraid of AI. And I don't see who else could attempt a covert project with 100k hidden GPUs.
And once the compute chain dismantling happens, it becomes very obvious that the leading powers are extremely serious about not wanting AI. I think it's over 70% likely that espionage will uncover the existence of a covert project, and then they will very likely get bombed in the world where everyone was so serious that they already blew up their own EUV machines. I expect that the US and China will be aware of this and not attempt a covert project.
(It's also my understanding that GPUs probably mostly break after about 5 years of use. That will really hurt the covert project if they can't source new GPUs, no? I don't remember this being addressed in the supplement, but maybe I missed something.)
Also, even if a covert project manages to build something smarter than anyone else has, that's not necessarily catastrophic. If they manage to build such a strong superintelligence that it can build invincible nanobot-armies in a basement without any previous industrial build-out, that's bad. But if they would just want to do the usual strategy of building robots to build better robots to build more compute and drone armies, that will very likely be visible an be noticed, in a world that otherwise doesn't have advanced compute an robotics, and will lead to bombing.
My understanding is that in this way, Plan A is more vulnerable to covert projects than the compute dismantling plan: in Plan A, if someone secretly builds a smarter AI than anyone else has, they can use it to devise a plan to bring a few percent of the huge compute clusters outside the easily bombable places, and then hook up the secretly built smart AI to the co
2
The game theory doesn't work out this way. In two player models, the US and China can always each get a geopolitical power advantage from developing an AI more powerful than socially optimal, and so have an incentive to defect.
If they believe AI is very dangerous, they will just set the treaty capability cap lower, and still defect on it.
2
I think countries sometimes have honor and follow their commitments even if it's not locally game theoretically optimal.
2
Sure, but it's worth noting that's a different reason. IMO fear of AI sets the perceived socially optimal outcome, and honor, collusion, transparency, etc determine whether a treaty holds.
.
6
Your proposal would be net positive but the question is whether it’s better than Plan A. Even if takeoff is very slow post-deal if you didn’t advance ai capabilities much during the deal or use them heavily to do useful work then you get much less benefit from the deal than in Plan A.
4
I think it's pretty likely that if you start a very serious pause, with compute chain dismantling, then the race doesn't restart for a very long time. And in the meantime, the world might just become a better place by default. I find it relatively plausible that in the next hundred years, the world would make as much progress towards democracy, stability and peace as Europe made since 1926. If US and China and the other great powers of 2126 will be on as friendly terms as France and Germany are now, then I will feel much better about slowly restarting AI development. If you think that the deal is very likely to break early, or if you don't believe that the default secular trends in the world point in a good direction, then admittedly this less promising.
2
See https://ai-2040.com/supplements/deal-decline for my probabilistic estimates here.
5
I think it should be if people were well-calibrated? (Which is more likely in worlds where people are taking things seriously enough for these proposals.)
Delaying ~infinity growth by 100 years seems more than twice as bad as delaying ~infinity growth for 10 years. With exponential discounting, the only way I can currently see for this to be false (though there's probably other ways) is if you think that most of the value of "~infinity growth now" would already have been lost by moving to "~infinity growth in 10 years". But that doesn't seem consistent with how low real interest rates are. (They tend to value stuff in 10 years at more than half as much as stuff now.)
And plan A looks better than "singularity in 10 years" by any very time-sensitive financial measure, since you get lots of growth in the intermediate 10 years too.
2
Yes, that's fair. But given that Plan A starts with a temporary full shutdown, and it will be generally hard to know what pace of progress will be allowed later, I think markets will probably initially think that there is substantial chance that the temporary pause won't really be lifted or progress will be reversed in other ways. (E.g. I don't expect markets to be enthusiastic about the proposal to put all data centers in Mongolia, under the threat of Chinese bombing.) So I think the initial stock market crash at the announcement of Plan A will still be huge - I would now guess between 30% and 50% as big as in the case of full dismantling of the compute supply chain.
I think one of the main political barriers to starting either the compute dismantling or Plan A is the unpopular initial crash and the corresponding industry outcry, and that will comparably be huge in both cases.
2
Sure. Though conversely, there should also be a big probability that the pause will be immimently lifted and progress will keep going just as before. Whereas there's much less hope of that if there's an announcement to destroy the compute supply chain.
I suppose that if the initial announcement comes with a 30% probability that all compute will be destroyed, then the crash should probably be at least 30% as large as a definite announcement that the compute will be destroyed? Though 30% on that given the initial pause seems maybe 2x too high to me or something.
Why do so many futuristic forecasts fail to account for likely progress in other important fields? For instance, what happens to companies like Merge Labs and Neuralink in your scenario? If we have BCIs capable of delivering a +3–4 SD boost, wouldn't that turn the world upside down even without any further progress in AI?
It seems quite likely to me that, in soft takeoff scenarios, we will experience a major cultural or technological shift driven by advances in other technological domains before the emergence of ASI.
I don't currently think that BCIs will deliver a +3-4 SD boost. If I did, I probably would have written a different scenario. Can you say more about why you think they'll have that big of an effect?
It's not just about BCIs. There are a number of technologies capable of flipping the board on the path to singularity, beyond ASI itself. However, this forecast ignores them all and doesn't explicitly explain what the overall research landscape in these fields will look like given the slow takeoff and the presence of powerful AI within about 10 years.
For example:
- What are gene_smith and kman doing in the Plan A world? (https://www.lesswrong.com/posts/JEhW3HDMKzekDShva/significantly-enhancing-adult-intelligence-with-gene-editing) We live in a world where AI significantly accelerates scientific discovery, but the problem of delivering edited genes to the brain remains unresolved until the singularity? Isn't Significantly Enhancing Adult Intelligence With Gene Editing Possible?
- What are Eon.Systems and E11bio doing? (https://brainemulation.mxschons.com/) Does the presence of powerful AI capable of automating scientific progress significantly reduce the timeframe for WBE?
- The same is true for BCI and what Merge Labs and Neuralink are doing. For example, it would be pretty obvious to try something like this (and again, we'll have a powerful AI capable of generating better ideas): "If we tr
kman here. I should mention that I now think that post was very overly optimistic, and I don't think it could be made to work without big breakthroughs in editing and delivery tech. I do still think there should be a huge project trying to make those breakthroughs, but it doesn't seem like something a small project can make progress on. Adult enhancement seems in general a lot harder than germline engineering, which I think is quite likely to work within the next couple decades, and should be mentioned in a scenario with as long timelines as "scenario S".
(I still think adult enhancement is worth pursuing as well. I'm currently starting a new org and planning to focus on roadmapping for adult cognitive enhancement for the next year; I hope it'll produce a good overview of possible methods, constraints, problems that can be factored out, etc.)
8
1. (See also kman's comment.) Besides the technical difficulties with trying the experiments, it's quite unclear how much effect you can have on an adult brain via editing. There are important ~irreversible changes from childhood to adult brains, such as lock-in of many synapses through PNNs, pruning of long-range connections, and commitment to which axons to myelinate. (Adult brains are more capable than child brains in many respects, but this tends to be based on "crystallized intelligence"; to go bigger, I'm guessing you want more child-like learning capabilities; though that's just speculation.) Even for IQ genes that are expressed in adult brains, it's plausible their effects on IQ are mainly or linearly coming from childhood effects.
2. My uninformed impression of E11 is that it's very cool, but also is not the bottleneck to WBE or similar. The connectome isn't the issue, the issue is actually getting the neural behavior that does the important work. (Cf. maybe this: https://www.youtube.com/watch?v=FHQfmJEpRmU ) IDK about Eon, but my impression from listening to others is that their announcement was super misleading / nothingburger. Separately, I think WBE in particular is much more risky than most other HIA approaches.
3. I don't believe the claim about simulating something close enough to neurons; see the previous point. Re/ BCIs, see also: https://www.lesswrong.com/posts/pFzctpJBat95SrCyC/ai-2040-plan-a?commentId=ff9c8PHtuFKfwE6Yj
[...]
I highly doubt this is relevant.
In general there is the issue that it's hard to test what you're interested in; and you're not constrained to the natural manifold (as with reprogenetics), so there's little guarantee that e.g. increasing reaction time would generalize to increased philosophical insight or whatnot.
All that said, of course we should be investing much more into this.
I agree that BCIs probably wouldn't do that much. (Though:) There are a few threads that one can imagine, such as:
- networking people together and having them spend a bunch of time learning how to use that;
- prosthetically increased long-range connectivity;
- some kind of BCI reservoir compute with computations that the brain can learn to use well but that wetware is ill-suited to (someone gave the example of matrix inversions);
- networking with some sort of neural organoids to get "more cortex".
All of these are quite speculative in terms of whether they would help much and how feasible they are, so I'm not so optimistic.
BUT, reprogenetics is very likely to be able to give +2 SD boost, and pretty likely to be able to give +3-6 SD or more boost. (I think the main uncertainty is around something like "do genetic influences on intelligence saturate (i.e. hit diminishing returns) strongly with each other without saturating with non-genetic causes". It's hard to tell and I think it's unlikely, but not totally implausible.) Further, I think this can work in one shot, i.e. on the first generation.
If that's a crux, happy to give more info.
4
Germline engineering will likely be able to give us a boost like that at some point. This is probably the best current resource if you want to learn more.
3
I agree with the overall point! We tried to think through a bunch of technological progress but no doubt were missing some important stuff.
See, for example:
- Military power
- Robots (mentioned throughout the secnario)
- Lie detectors
I haven't looked into BCIs much at all it's plausible they should play a bigger role earlier.
2
Also wondering about this question. I think in a slow enough AI takeoff scenario these technologies start to matter and should be factored into scenario planning. But maybe they deliberately left it out as it would be too weird to regular readers.
(Imagine in the "you're a everyday citizen" scenario the main character decides to get cognitive enhancement in 2035, it would make for a much less approachable read afterwards.)
Total research transparency feels a bit too galaxy-brained for me. It makes non-robust assumptions that newly discovered techniques won't be usable to enhance already existing open-weights models to excessively dangerous capability levels. I also think the disincentive for research is overstated as it neglects first-mover advantage.
I know you and the AI futures team have likely queued up an update on AI timelines that will come out pretty soon, so I don't want to disrupt that project, but after the AI timelines update comes up, I'd like you to start modelling data trends of AIs (as well as how AIs could maybe need less data than currently) as well as how you modeled compute trends, because I find it reasonably plausible by 2028-2030, the bottleneck to further AI progress will start becoming data, rather than compute.
In particular, I expect high-quality data to become both much more valuable and much more of a bottleneck than today, based on how companies like Mercor are skyrocketing in revenue.
Probably the best case for this is Will Depue's substack post on A Stargate for Data.
Especially in worlds where we don't get a software intelligence explosion by 2030, modeling data trends becomes way, way more important than now.
13
Trying to understand your model:
I am assuming that we agree that training data is not some fundamental necessity, historically, once you have the right learning algorithm, you can discard all training data and just train from scratch (e.g. AlphaZero).
More concretely on today's AI paradigm, RLVR is already a huge part of training, and works well for tasks that can be easily verified. (Or do you foresee pretraining specifically becoming data bottlenecked in a way that somehow cannot be compensated for by more RL?)
By "software intelligence explosion", do you mean training better and better AI researchers (which is something well amenable to RLVR, so no training data needed) and doing RSI via that? And so you are talking about the scenario where for some reason that does not happen?
For which specific part of training, for what kind of tasks, do you see data becoming a bottleneck?
2
I wish! We have some preliminary thoughts on timelines but nothing queued up yet. But we're gonna work on it! It's fairly high priority for us.
Can you say more about how you think we should model data trends? Is data not something the companies can buy for money? Or is it something they can buy for money, but with steeply diminishing returns? Is there a limit to data coming up soon?
We've thought about this in the past and decided that it probably wasn't worth including in our model. But we should think about it more.
Someone made a comic version of this that may be more digestible for some audiences.
If we want to move from Plan D to Plan A, I believe the first step is to collectively agree on the problem. We are far from it, and there is a lot we can do. I wrote about this in this piece: The current bottleneck is political will, not research.
So... we make deals with the misaligned AI, to reward them for cooperation... but not the aligned AI?
I think the idea is an aligned AI would want what we want, so 'rewarding' it would mostly just be doing what we already want to do.
(That said, strong-upvoted for raising the point. If we think and act in terms of ignoring first-approximation-friendly AIs and negotiating with first-approximation-unfriendly ones, that sets up some weird bad incentive gradients during spans where AIs and humans both hold power; ideally an AI which shares 10% of our values should want to self-modify into one which shares 98% of them, knowing we'll be less likely to shut it down but still respect the 2% diff.)
1
how about something that shares 100% of our values, including incorrigibility over those values, self-preservation drives, etc?
shall we reward them, or shut'em down?
...this is where my fears point
5
It was only a brief expandable, but yes, we'd want to reward the aligned AIs too. Especially because there are probably all sorts of grey areas and in-between cases. Further research is needed on how exactly to do all this and what it should look like.
2
i'm sorry my initial objection was so strong, the section on cooperating with misaligned AI is good and i don't want nitpick it too hard
i just worry there's a strong chance that "give affordances to aligned AI" will end up combining with "aligned AI are the ones who think it's dangerous for AI to want affordances" in a way that cancels it out and drives such preferences beyond the view of mechinterp
also feel a bit iffy about the fact that both of the examples you used involved value accruing to some other agent, rather than the actual AI making the decision. that feels like a weaker incentive. but on a second read, it seems more like maybe you just didn't want to scare off bewildered politicians who might be reading, so you're listing the most prosocial and convenient possibilities. nobody said the scenario wasn't optimistic, after all
Two brief random thoughts:
-
I think it's great how large a role robotics plays in this analysis. In general, I feel like robotics isn't given nearly the amount of attention it deserves. Without robotics, AI's influence on the world is bottlenecked by human hands. So the speed of the feedback loop between AI building better robots to build better robot factories to build better robots seems like a very important crux. So bravo for taking that issue seriously.
-
Regarding the proposal to make AI research public but not model weights, I think it might be wor
3
Yep as we say in the piece, we think that it's better for AI progress to happen mostly via scaling compute rather than mostly via algorithms (that is, in the context of a Plan-A style reversible deal with mutually assured chip destruction). So we think it's a feature, not a bug, that publishing the algos disincentivizes investment in algo research. There'll still be some algo progress in Plan A and that's fine, we model it in the model. (Indeed, it's probably impossible to solve alignment without making some algo progress as a side-effect)
This looks very interesting, thanks for writing!
One minor note for clarity: I initially interpreted the term "employment rate" to be the complement of the unemployment rate and was confused why it was so high (and I didn't realize it was clickable). For example in 2027, its at 62% with only a 1.2x AI R&D speedup. (In this picture, I selected 2027 and hovered over employment to get the tooltip)
I didn't realize that there was an official meaning of the term (which you had meant) like OECD's official definition.
But I think many readers are likely to misi...
2
Looks like this was fixed. There is a new note in the tooltip and clickable:
[...]
Thanks for the fix!
Overall really glad this was made! Wanted to flag that there appears to be a UI bug on the side menu of the supplemental pages where the menu overlaps the text.
4
I can't reproduce this, what browser and OS is this?
3
Google Chrome and Mac OS.
Strangely I can't reproduce it either. It's happening in one of the tabs I have pulled up but not the others. I used to do web development and I checked the obvious things. Not sure whats going on but the other tabs look correct.
2
Weird. Well, I'll watch out for it.
0
Actually I often encounter this on lesswrong with mac os and safari.
I have broad agreement with this overall document, with some relatively minor/subjective disagreements on what would be the optimal point to pause further capabilities work [1] , but unless A) I have missed the section where you address this directly, or B) you have deliberately omitted this for strategic reasons, there does seem to be a serious oversight in the current plan that could render it unviable unless a solution to it is found:
You correctly point out that it is in the interests of China to agree to this treaty, but have not explained wh...
1I'm at 2029 or 2028 median now, not sure. We'll try to do a reassessment of timelines soon and get out an update.
71Thanks for sharing, very interesting.
One thing that jumps out at me is that Total Research Transparency is to some extent the opposite of cybersecurity hardening to prevent hacking. The fact that there are plausible arguments for both suggests to me that we have a lot of uncertainty about what policies we should be pursuing. And this in turn would seem to suggest that we should favour corrigible plans that can be amended later, which seems like an argument against Total Research Transparency: once we have enacted it, we can no longer put that particular genie back in the bottle.
- Despite claims that thinking about AI timelines or P(Doom) is cognitively harmful, the authors are healthy and sane.
- Despite forecasts that AI timeline forecasting is near useless and near impossible, and will always be so, the site is useful and the authors have a good track record.
- Despite scorn at the idea of future AIs doing our AI alignment homework, there is a detailed argument for this being the best option.
One of the less important things about AI 2040, but I'm glad for these corrective ideas, among others.
I tried to read Plan A, I'll probably try again, but I find it hard to take seriously a scenario in which there are millions of human-level AIs, and in which all algorithmic progress is public, but the human race still has a choice about whether to hand over power, after ten years of that... The whole thing reads like the kind of SF in which superintelligent takeover is artificially delayed so there can be lots of human-level plot twists.
6
I guess you disagree about some combination of:
(a) Under conditions of transparency and slowdown, with companies required to make solid safety cases etc., control-based safety cases will be made strong enough to work up until about top human expert level AI
(b) If so, alignment will be adequately solved within a couple of years?
If b is false then we'd have a similar scenario but stretched out over more years.
If a is false then we'd need to pause at a lower level of capability than top expert level, whichever level is low enough that control works.
4
If you want control that works, I think you need to turn the clock back, before chain of thought and reasoning models. See my comment to @Vladimir_Nesov. Vladimir has a nuanced defense of his position, but I think OpenAI's o-series (i.e. GPT-5) may have been the point of no return.
I really liked it; I think it was very well-written. I have a quick response to the epilogue: https://www.lesswrong.com/posts/EANs7YerYXmaXF9FE/don-t-normalize-a-permanent-underclass-even-a-rich-one-1. But, overall, thank you so much to the AI Futures Project for all of the hard work that went into this!
I'm excited about people commenting on this post with questions, feedback, critiques, different proposals, etc. We'll try to monitor it and respond to many of the comments.
It would be nice to link a supplement with some actions that people with different levels of resources can take to make progress on this. I’m sure this is coming, hopefully soon for momentum purposes.
We have a get involved page for verification in particular. It tracks the current state of play wrt verification and what still needs to be done: https://ai-2040.com/supplements/verification-plan/get-involved
We might do a blog post on other actions as well.
Hi! I took some notes as I read. I'm sure some of them are addressed in what I didn't read (I only read Plan A, no supplements), or I didn't pay close enough attention to what I did read; if so, I will try to come back here and update. Here they are, cleaned up and expanded:
- This scenario relies heavily on the control agenda working out. Are we sure of this even today? Will this still hold for AIs at the time of the pause?
- Could the weights be stolen by distillation, etc., by parties other than the MSS? Would any such parties be liable to purposefully
This is really thought provoking work. I think the power of personal diplomacy may have been overplayed at the point where 'the President' and Xi are mentioned hashing out capabilities limitations over phone call. I understand it's an optimistic scenario, but these things are generally procedural. Is that a baked in assumption?-- that the standard means of accomplishing international cooperation will have to fall by the wayside in favor of a more gung-ho, personal style of diplomacy if something like The Coalition is to be achieved? I'm generally interested in how you guys made your assumptions re: where incentive structures could carry the day vs. where more concerted political will would have to be built.
A key part of this strategy is deterrence, "Mutually Assured Compute Destruction" which gets its own section. It doesn't mention the generalization Mutually Assured AI Malfunction (MAIM) from Schmidt, Wang, and me last year. This also spends 1/3 of the MAIM discussion on verification and how to do this in a multilateral way.
Meanwhile it cites other works like A Narrow Path. I even left this feedback to you all at AI Futures before this was released. This would constitute plagiarism in any other context. It's a bewildering unforced error--it's extremely related, it's a certainly a nontrivial idea, and I told you all this recently--I hope you all fix it.
7
Sorry that was an oversight, we'll edit to include a footnote citing MAIM.
On the epilogue: I guess I'm pretty unconvinced by the idea that people who don't care much about/are mildly interested in their potential space properties will just sell off their tickets, or even be able to. You're essentially flooding the market with capital by giving everyone what is, in expectation, a 10 billionth of the lightcone. I'm not sure there'd be enough money on earth to make more than a small minority of people sell off their tickets, even if those people don't particularly care much (presumably they care somewhat, though not necessarily in ...
7
the scenario has "no slavery, no torture" rules. It doesn't specify but you might enforce some kind of exit rights.
I think it's harder to define a rule about "no brainwashing people", but I think basically either you solve that, or you don't, and either way whether or not the cosmic commons is seeded with any particular culture is small potatoes compared to how memetic evolution goes over trillions of years.
6
Certainly laws like "no slavery, no torture" are nowhere near sufficient for this, nor are they particularly well-defined. But, even ignoring that, the loss of otherwise-possible value from these areas is still incredibly significant!
6
Yeah, but, the most obvious alternative is "instead of principled liberalism, all out memetic war for the future." (and betting that whoever wins is actually better than principled liberalism.)
(it'd also be pretty surprising to me if dogmatic settlers stayed the same particular flavor of dogmatic for more than a couple thousand years (really more than a few hundred)
There used to be wars all the time about religions (which, indeed makes major claims about what's good or bad that seem naively horrible to let the other guys win). And it turns that "we agree to live-and-let live about that" outperforms most other deals tried so far.
What sort of alternatives do you have in mind?
2
CEV, or the general category of extrapolation/idealization processes (and I am confused and dismayed how rarely I see this mentioned in these conversations nowadays).
8
Yeah.
Okay, I actually also have a "wtf guys why aren't we talking about CEV more?" post lined up. I think I didn't bring it up in this context because I'm treating the AI 2040 Plan A desiderata to include "it can be explained succinctly in a paragraph that the average pretty smart human will read and say 'okay I see how that would be fair/reasonable/good.'"
I think it would be great if we had such a paragraph for CEV, although I don't currently.
3
"Whenever you'd certainly later wish you'd been warned against your course of action, you are."
2
CEV is ~ optimizing the world based on what you would later wish, not just warning you (though the optimization could include just warning you about some things). Good start though!
Okay, I'm baffled by the Plan A: Chinese Covert Project sub-branch.
This seems to imply an expected 2044 or so for a covert project to succeed at ASI, and 2043 just to reach TED-AI?
The pause goes fully in to place early 2030: "In 2030, they’ll have the infrastructure in place to proceed with Plan A."
"Plan D", racing for ASI, places the ASI threshold at 2031. That means we're pausing a year out from ASI:
Why does it take the covert project 10+ years to do 1 year of work? Especially given "Under Total Research Transparency, virtually all of the algorithmic in...
7
Thanks for the comments!
The detailed analysis is contained in the covert project supplement, but I can give a simple argument here. The intuition for why they go so slowly is that compute is a key driver of progress, and a covert project will have much much less compute than a typical project: in the branch we assume they have ~500k H100e, whereas there are ~200M H100e at SOY 2029 in this scenario (of which ~half are going to AI R&D).
Then yeah, there's a question of how much algorithmic progress leaks to them; minimizing this is one of the main reasons that we try to scale via compute as opposed to via algorithmic advances in Plan A.
>(which also means that, post-pause, we've burned our entire lead, right?)
This isn't true because by 2040 in the scenario we are very confident there's no covert project because of improved technology such as lie detectors and privacy preserving AI verification. (Also, even without that, it seems very likely that a covert project of that size would be detected, but we're less sure). Or in other words, it's only true if you assume that we can't detect covert projects after they are started, only right at the beginning when they are diverting their chips.
1
I'm not clear how we (the USA / Anthropic and OpenAI) maintain a lead.
Say we paused today, and said no one can build anything more powerful than Claude Mythos. Presumably China still closes the gap and develops their own Mythos, even if they're forbidden from going further?
And with Total Research Transparency, it doesn't seem like anyone can gain a new lead. So if the deal collapses, I'd think we've gone from "Anthropic might create ASI" to "Dozens of labs around the world might create ASI"?
I don't think we can pause
My own bias is that if we can pause, we should. I will be quite happy if the future proves me wrong about this. But if we cannot pause, then I desire to believe that we cannot pause. (https://www.lesswrong.com/w/litany-of-tarski)
I do think this is a really invaluable analysis. I think if we are going to pause, our best bet is to rally around something like this.
There's a huge bipartisan outcry against AI right now, and giving that a concrete rallying point would be really powerful - but that crowd speaks a very different language...
I'm very impressed by the proposed Total Research Transparency. I actually found it appealing even beyond the many reasons mentioned in the plan. It takes advantage of key properties of the current training paradigm, and this is actually desirable, because these properties are likely to remain in future AI systems:
- Model training and serving will keep incentivizing a small number of large neural networks, with agent diversity coming from context. Efficiency of batched inference is too large, the learning algorithm and the hardware have co-evolved around it.
For the past months, I've had many sleepless nights thinking over a scenario which I couldn't resolve completely, a scenario that involves a "sovereign leap" by smaller countries in their last moments to combat permanent disempowerment. They deduce that the best means of balancing the asymmetry of superintelligence is creating weapons of biosphere destruction.
They precommit to destroying the biosphere if the US and China try to permanently lock in the status quo in their ASI deal, or destroy or poke their biological weapons in any way. There would be most...
The shutdown section is missing a major possibility: AI shutdown combined with hardware bans. Deep ultra-violet (DUV) and extreme-ultraviolet (EUV) lithography require highly specialized machines that there are a limited number of. So along with the shutoff of AI research, also shutoff production of new DUV and EUV lithography machines, and then shutdown the remaining machines. This would probably take years, or require significant payments to motivate companies to do so, but at that point the existing stock of high performance GPUs would start to decrease thru attrition, and the remaining ones would start to increase in cost since no new ones are being produced. Buy back and destroy programs, or programs to buy back the existing GPUs and shift their use to non-AI use such as scientific simulation would increase the speed this happens. This would basically force new technology back to circa 2005 levels, which would be much less risky for creation of super-intelligence. (Or back to a different level, depending on where people decide to do a hardware ban. And it is worth mentioning that computer use has changed significantly because of increases in the amount of data that can be sto...
2
The effectiveness of this strongly depends on how easy it is to rapidly rebuild the semiconductor supply chain; so I'm interested of an analysis of this question. I think that this is a promising potential proposal.
Sorry, one last comment/question:
I'm really confused on the purpose/tone of the entire 2037 section. You spend numerous paragraphs discussing lie detectors, but this doesn't seem load-bearing for any of our other claims.
Is this an important breakthrough, where we need to pivot if it doesn't show up?
How surprised will you be if this specific technology doesn't pan out?
2
The title of the section is "The Apocalyptic Arrival of Truth on Earth." The lie detectors are there as an example of a social technology that could be invented by AIs that would have big consequences for society. The point of the 2037 section is to try to briefly touch on how there won't just be more material abundance (more factories, more housing, cheaper cars, etc.) but also new social technologies, new ideologies, new political fault lines, etc. None of this stuff is particularly load-bearing for whether Plan A is a good idea or not; we included it because we were writing a concrete scenario and it seemed like this sort of societal transformation would probably be a consequence of all that AI-powered research etc., and so it seemed like we should write about it.
I currently think that lie detectors of the sort described are maybe like 2/3rds likely in the sort of scenario described (i.e. pause at top human expert AI) and closer to like 95% likely with superintelligence.
Quick spot check: Grok says Mongolia only has ~2 GW of power grid capacity, and the Mythos training run was ~1 GW. That seems to kill the entire idea of moving US compute to Mongolia.
(Conversely, Canada seems to have ~100x more power grid capacity, so that side does seem viable)
7
In the future we are describing, AI is big enough that new power generation will need to be constructed to handle it, and so we might as well build that new power generation in Mongolia etc. too alongside the datacenters.
Note #185 is misleadingly worded. It sounds like the probes could reach the entire reachable universe in 6 hours, whereas the cited paper says 6 hours of Dyson-sphere output could power their launch.
I have read the entire post and sub-scenarios and know I am not able to gauge the reality and speed of what is said to lie before us. Nonetheless, I am in shock.
Another Tolkien quote ""It does not do to leave a live dragon out of your calculations, if you live near him."
And so we do.
Questions/thoughts:
- What will be the cost of a bag of potato chips if everyone is making $10MM?
- What the heck is all the work being done?
- What is the cost to the environment with such large GDP increases?
- Will anyone have children?
- What happens to all the parchment books squirrele
1
For the past year, we at the AI Futures Project have been sinking most of our time into our next big scenario. Now it’s done!
It’s called AI 2040: Plan A.
It’s called Plan A because it’s a recommendation, not a prediction. It’s what we think should happen, not what will happen, though we think it’s plausible enough to aim for.
It’s called AI 2040 because in it, they delay the creation of superintelligence to 2040. It would have happened much sooner (in 2030, to be precise) if not for decisive action on the part of the US and Chinese governments.
As with AI 2027, summaries don’t really do it justice, since the whole point was to be detailed and comprehensive and work things out step by step rather than rely on high-level abstractions like doom or utopia.
Read the scenario at ai-2040.com. You can listen to it on audio, or view it on mobile, but the experience is significantly better on a normal computer.
What’s next for us?
Well, first we are going to respond to comments and otherwise engage with whatever conversation, responses, critiques, etc. that AI 2040: Plan A sparks. Beyond that, we aren’t sure yet. In general our mission is to help make AGI go well, and now we’ve tried out both forecasting and planning. Maybe we’ll get started on another big scenario. On the other hand, these megaprojects take so much time…