Yeah, the thing the 'scaling extrapolation' view doesn't take into account is that as soon as radical speed-ups to algorithmic research are made possible by AI R&D agents, suddenly the trendlines for algorithmic progress should be projected to steepen. How much and for how long before slow-downs are hit? That's unclear. I think there is at least some substantial probability that no slow-downs are hit before full AGI, and some smaller but still considerable probability that the improvement cycle rushes forward at high speed past that point to ASI.
This should be assumed to potentially involve dramatic gains in both peak capabilities, and in efficiency and speed of training and inference. If so, then compute governance becomes completely irrelevant for blocking creation of dangerously powerful AI. It can still help put limits on the amount of inference used. Why? Because no matter how efficient the AI is, if you have more compute you have more parallel copies (and can run them faster up to the limits of the system, which is probably somewhere between 100x to 1000x human thought speed).
If we are going to head this off, we need new governance methods, and soon. Maybe really really soon, like, before the end of 2025. Hopefully we have until more like 2028, but we can't count on that for sure.
I have very little faith in current governments to implement and enforce policies that are more complex than things on the order of governance compute and chip export controls. Much less to do so within the short timeframes we are facing.
I think the conclusion this points towards is that we need new forms of governance. Not to replace existing governments, but to complement them. Voluntary mutual inspection contracts with privacy-respecting technology using AI inspectors. Something of that sort.
Here's some recent evidence of compute thresholds not being reliable: https://novasky-ai.github.io/posts/sky-t1/
Here's some self-links to some of my thoughts on this (I recommend reading the posts these comments are on as well):
https://www.lesswrong.com/posts/tdrK7r4QA3ifbt2Ty/is-ai-alignment-enough?commentId=An6L68WETg3zCQrHT
[Epistemic status: I'm trying to make my thinking legible to myself and others, rather than trying to compose something highly polished here. I think I have good reasons for saying what I say and will try to cite sources where possible, but nonetheless take it with some grains of salt. As with my last post, I am lowering my standards so that this post gets out at all.]
I'm working my way through Harvard's AI safety club (AISST)'s modified version of the BlueDot Impact AI Governance Curriculum. I'm doing this because I am pessimistic about technical alignment on current AGI timelines, and so I am trying to extend timelines by getting better at governance.
I've already taken MIT AI Alignment (MAIA)'s version of the BlueDot Alignment course (MAIA's version is not publicly available), and I've taken MIT's graduate-level deep learning course, so I'll mostly be skimming through the technical details.
The purpose of this sequence is for me to explain what I learn, so that I internalize it faster, and so that I can actively discuss with people about my takes on the readings. Reading this is not intended as a substitute for actually doing any of the above-mentioned courses, but it might give you some new ideas.
EDIT: These posts were taking up too much time, and were serving as a blocker for me actually learning the content, so this post is discontinued. If you want my thoughts on anything specific in the curriculum, feel free to message me.
I'm already familiar with a lot of the content by osmosis and by my more technical AI safety background, so this post will probably be shorter than some of the later ones. Even though the curriculum is organized into weeks, I don't plan on doing these posts weekly. I will do them as fast as I can, given my other commitments.
I've watched this video a handful of times in the past, and since then I've gotten significantly more technically skilled at AI stuff. I'm gonna skip out on this one. If you haven't seen it, it's an excellent explainer.
Technically the curriculum only says to read the executive summary. This is once again some basic technical stuff that I mostly skimmed through. The triad it describes is compute, data, and algorithms. Those are definitely important things, now what about them? In the rest of the first section it goes on to define other stuff like "machine learning" and "supervised learning" and stuff like that.
In section 2, it talks about how the three elements of the triad can serve as levers for policymakers to control AI development. In the context of the data, we see two main focuses:
In the context of algorithms, they talk about talent pipelines, visa control, and worker retraining, mostly from the context of doing capabilities research. I don't have strong priors on this, but bringing in a lot of extra technical skill seems like it will help capabilities more than it will help safety, by default. Still, weak overall opinions here.
In the context of compute, they mostly talk about supply chain regulations. This seems straightforwardly really important for regulating scaling, although I feel like they're probably missing some other parts of compute governance.
Overall, even though this paper is mostly focused on capabilities research, it talks about some useful policy levers. I think I already had a decent number of these concepts in my head, but it's good to make them more explicit.
This article shares some pretty standard graphs and quotes, mostly from Epoch, as well as one from ContextuaAI. While Epoch doesn't directly forecast AGI timelines here, I think these are still pretty important for showing that things are just continuing upwards. Seems sound and correct.
This gets into some really good stuff about chip manufacturing that I mostly didn't know before! They get really into the weeds with TSMC and NVIDIA numbers, which I won't copy here. The tl;dr is that they forecast an increase in compute of the largest training runs of about 4 OOMs by 2030, with some decent-sized uncertainty given the large number of constraints playing together. Figure 1 has a really nice explainer of their different estimates of important constraints (check it out on the website, since it's interactive and has multiple slides).
I think their mainline prediction should really account more concretely for the unprecedented economic growth that AI is going to bring over the next few years, and the unprecedented demand for AI chips that this creates by default as soon as this growth becomes widely apparent. Their predictions are being very conservative in this respect, and mostly not accounting for AI speeding up economic growth as far as I can tell, nor are they accounting (in their mainline prediction) for weird discontinuities in demand to TSMC as people realize just how big AI is becoming.
I don't know how I made it this far without much understanding of the synthetic data generation process, other than "just have the model make data." I'm a bit disappointed that this paper doesn't include synthetic data in their prediction of dataset growth, but I understand that they don't have robust ground truths to base their predictions off of, like in the other domains they investigate. However, this is another reason to suspect that they are underestimating the trends, since synthetic data will likely play an important role. Since LLMs are much better at evaluating the quality of data than generating high-quality data, they can just generate a bunch of raw synthetic data and filter it. They mention concerns of model collapse due to too much synthetic data, but once again don't incorporate this. I think that, in the age of o3 and other thinking models incentivizing companies to go hard on getting a lot of compute, it might be a lot easier to get a lot of synthetic data using their large clusters while they're not actively training models. It seems like the straightforward way to turn an excess of compute and a bottleneck of data into a balance of the two.
However, the paper doesn't predict that we'll end up in a low-data and high-compute scenario, given the other concerns about compute supply chains, but it doesn't rule that situation out either. It predicts that energy and compute will be the primary bottlenecks. I think that both of these are flexible given the economic upset that I hypothesized above. Their conclusion: the 4x training compute increase per year can likely continue until at least 2030, and labs are incentivized to do so.
Energy bottlenecks seem the tightest, and the most feasible scaling strategy is to build data centers in a lot of different places so that they draw on different power grids. This is apparently worth the latency, which seems reasonable.
I've heard a lot already about compute governance, talent pipelines and the like already. One thing that the last reading revealed as possibly important is energy governance of AI. Maybe people are talking about this and I'm just not hearing it, but if that's the tightest bottleneck on development, then it's a powerful lever indeed. These sources feel a bit outdated, since the landscape has massively shifted even in the time since these articles have come out (o1, then o3 and deepseek v3). I don't think we know how much compute o3 took to train, but it's giving me the impression that OpenAI pushed above the trend line in terms of all the different things we're trying to predict here, and so we have to adjust further.
I still feel like all this talk of "scaling to 2030" is a bit misguided, since I'm ready for AGI to be here sooner than that. It is, however, further evidence that we're probably not going to run out of resources before AGI.