Thank you to @Thane Ruthenis for discussing this topic with me previously. Some arguments in this essay are taken from that discussion, but all mistakes and/or bad arguments should be attributed to me. You can view the original discussion here (optional for reading this essay). Thank you as well to @grainsilo for feedback on an earlier version.
Consider the following argument for discussing AI capabilities as a safety-focused person:
Understanding AI capabilities better has many strategic benefits, particularly for timelines.
Yes, good ideas tend to break containment, so some of our AI capabilities analysis may get read by capability researchers…but at the same time, these people have already thought about these questions more deeply and thoroughly than we have.
Therefore, none of our arguments are new to capability researchers.
Therefore, we will have no impact on advancing AI capabilities or timelines by discussing this topic.
I’ve seen this argument made explicitly in some places, I presume it's made in many people’s heads whenever they post about capabilities, and I’ve certainly made this argument to myself before.
I now also think this argument is wrong. I don’t really know what to do about this conclusion. Maybe the answer is nothing, but I don't know that, so it seems like a good idea to make people more aware of the reasons why I think this, in the hopes that it prompts further discussion, reflection, and caution.
Problem 1: High-Surface-Area Questions
Certain fields/problems/questions can be aptly described as being “high-surface-area”: complex, with many interlocking pieces, lots of potential connections to other fields/problems/questions, layers of better abstractions/concepts/ontologies waiting to be peeled back, and multiple plausibly helpful (although not necessarily ultimately helpful) lines of thinking that can be pursued with respect to the question.
Some discussion of different approaches to the problem and their feasibility here
Basically any problem in political science, economics, or sociology.
Prospect theory as an example of how a new line of thinking revolutionized a field (in this case economics)
It's hard to analyze virtually any political question on its own, example from Beware The Man of One Study: “the minimum wage probably works differently depending on what industry you’re talking about, whether it’s state wage or federal wage, whether it’s a recession or a boom, whether we’re talking about increasing from $5 to $6 or from $20 to $30, etc, etc, etc.” I would add: works differently depending on demographic makeup of the area, work culture trends, season; whether you care about just jobs lost or also hours cut; whether you care if some demographics are affected worse than others; whether you care about working conditions… you get the idea
The interaction between various strategic questions about the alignment problem, including timelines, takeoff, alignment difficulty, etc
Many mathematics questions (particularly the most interesting ones) are often high-surface-area, as evidenced by some of them being solved by connections between fields, or solved only after solving some other seemingly unrelated problem, or solved only after creating some new way of thinking about the problem
And then our topic for today: the question of how to build AGI
With high-surface-area problems, even if someone has thought about the question for longer than you, you can still reach certain insights before them by thinking along a different track. This could be because you have a better way of attacking the problem, because the other person is using the wrong concepts or ontology, because insights from other problems/fields/questions can be brought to bear on the problem, and/or because the problem is just very large and it has many aspects to explore.
Because AI capabilities questions are often high-surface-area, it's not safe to assume that your analysis of them is day-old newsjust because you aren’t a capability researcher, or don’t work at one of the major labs, or haven’t read many papers on the latest developments in the field.
Making matters worse, I suspect that there are substantial differences between the minds and thinking of capability researchers and the minds and thinking of safety researchers, due to things like selection effects, what types of background readings are consumed as canonical material, and socially motivated reasoning (example). Because of the nature of high-surface-area questions, a good way to make progress on them is to increase the temperature/diversity of your or your group’s thinking. I worry that we are essentially providing this service free-of-charge to the labs.
This is mostly a tail-risk problem. The vast majority of discussions of AI capabilities by people on LessWrong are unlikely to lead to any insights towards creating AGI. But because of this aspect of AI capabilities questions, it's not as unlikely as you would initially think.
Problem 2: Discussion-Ecosystem Effects
I think many people assume (myself included, at one point) that if they make sure to only repackage and reorganize arguments about capabilities already made elsewhere, their analysis won’t have any negative effect. Contra this idea, Thane made the following points in our earlier discussion:
The biggest issue here is not even any particular talking point or thinkpiece, but the overall ecosystem of discussions on the topic. Such ecosystems prime more and more people to think about these things in the shower or whatever/have it as the top idea on their mind...
...arguably the danger is in feeding the discussion-mill around the topic at all, since it'd prompt other people to have more discussions and thoughts about the topic, illuminating the conceptual space further and creating fertile grounds for the sort of speculations and progress that are very bad.
Ok, but so what? Again, capability researchers already have these sorts of discussions and thoughts about the topic all the time. They also certainly have capability questions as their top-of-mind idea a fair amount of time. Alignment researchers discussing capabilities won’t change either of these things. This is true, but it leaves out two important threat vectors.
First, feeding the capabilities discussion-mill inside the safety research ecosystem is a great way to set up less careful safety researchers to make the high-surface-area mistake described above.
Second, while capability researchers are already keyed in on discussing and thinking about how to develop AGI, you can still affect how they are discussing and thinking. This relates to the points above: there isn’t just one line of thinking that one can pursue about AI capabilities, and you can help someone along by bringing their attention to better arguments and vectors of investigation. Presumably, when you repackage and recombine arguments about AI capabilities, you do so because you think those arguments are important and more people should be paying attention to them. Unfortunately, this ends up as a good signal to capability researchers as well. Here, you’re not really prompting other people to have more discussions and thoughts about the topic, but you are prompting them to focus their discussions and thoughts on the particular subtopics and arguments you think are most important.
What should be done?
It would be easy if the answer to this problem was to simply stop all discussion of AI capabilities. Well…maybe that is the answer, but I’m really not sure. The value of understanding capabilities for strategic purposes is pretty high. So I don’t really know what to do. Here is an assorted list of some parting thoughts, but I’m mostly leaving this as an open question for now (more ideas welcome!):
All these worries are moot if there are actually 0 major insights needed to get to AGI. This seems to be quite a popular opinion nowadays, but I’m not so sure myself, and I’d rather be safe than sorry.
On the flip-side, it could be the case that AGI is so far off that these sorts of discussions have no impact at all. But again, I don’t know that. We could be in sudden death.
In this essay, I’ve operated under the assumption that the kinds of capability arguments that are best for gaining strategic insights for AI safety are perfectly correlated with the kinds of capability arguments that are best for, well, actually developing AGI. I think there is some correlation, but it's certainly not perfect. Nor is it the case that every answer required for strategic clarity necessarily rests on questions of capabilities. There are very likely arguments or methods of discussion where you can get 80% of the upsides with only 20% of the downsides.
I don’t know what these are, though, and even figuring this out illuminates conceptual spaces we want to keep in the dark: determining which lines of thinking are most helpful for AGI development in general. But maybe we can know it when we see it, i.e., just use good judgment here?
Or maybe it's the case that all of the best 80/20 arguments have already been made (ex: here for long timelines), and only infohazardous ones can eek out more strategic clarity.
On the other hand, even if we did figure out (2), we would still have problems with discussions drifting into adjacent infohazardous territory.
The effects of everything I just discussed could be so small as to be negligible, but I’m not confident that is the case, and there seems to be a lot of tail risk here.
The type of capability discussions with the public and with policymakers required to advocate for a pause are basically completely fine in my opinion, as these tend to be more surface-level with respect to capabilities than, say, timeline debates on LessWrong. And they are pretty damn important.
There are plenty of people outside the safety scene that are willing to openly speculate about AI capabilities, so even if we decided to pull back on this a bit ourselves, I don’t think it would leave us completely in the dark.
I think the AI safety community would do completely fine if we never learned anything more about e.g. timelines, as long as we hedged our bets appropriately: invest in strong long-timeline bets such as HIA, work aggressively for a pause in case timelines are short, not do anything stupid out of desperation, etc.
I would feel pretty happy if the main takeaway people got from this post was to just reflect a bit more on the above problems before engaging in the next capabilities discussion they get tempted by. Simply being aware of these issues and incorporating them into our judgement is possibly the best we can do right now.
Thank you to @Thane Ruthenis for discussing this topic with me previously. Some arguments in this essay are taken from that discussion, but all mistakes and/or bad arguments should be attributed to me. You can view the original discussion here (optional for reading this essay). Thank you as well to @grainsilo for feedback on an earlier version.
Consider the following argument for discussing AI capabilities as a safety-focused person:
I’ve seen this argument made explicitly in some places, I presume it's made in many people’s heads whenever they post about capabilities, and I’ve certainly made this argument to myself before.
I now also think this argument is wrong. I don’t really know what to do about this conclusion. Maybe the answer is nothing, but I don't know that, so it seems like a good idea to make people more aware of the reasons why I think this, in the hopes that it prompts further discussion, reflection, and caution.
Problem 1: High-Surface-Area Questions
Certain fields/problems/questions can be aptly described as being “high-surface-area”: complex, with many interlocking pieces, lots of potential connections to other fields/problems/questions, layers of better abstractions/concepts/ontologies waiting to be peeled back, and multiple plausibly helpful (although not necessarily ultimately helpful) lines of thinking that can be pursued with respect to the question.
Some examples of high-surface-area questions:
With high-surface-area problems, even if someone has thought about the question for longer than you, you can still reach certain insights before them by thinking along a different track. This could be because you have a better way of attacking the problem, because the other person is using the wrong concepts or ontology, because insights from other problems/fields/questions can be brought to bear on the problem, and/or because the problem is just very large and it has many aspects to explore.
Because AI capabilities questions are often high-surface-area, it's not safe to assume that your analysis of them is day-old news just because you aren’t a capability researcher, or don’t work at one of the major labs, or haven’t read many papers on the latest developments in the field.
Making matters worse, I suspect that there are substantial differences between the minds and thinking of capability researchers and the minds and thinking of safety researchers, due to things like selection effects, what types of background readings are consumed as canonical material, and socially motivated reasoning (example). Because of the nature of high-surface-area questions, a good way to make progress on them is to increase the temperature/diversity of your or your group’s thinking. I worry that we are essentially providing this service free-of-charge to the labs.
This is mostly a tail-risk problem. The vast majority of discussions of AI capabilities by people on LessWrong are unlikely to lead to any insights towards creating AGI. But because of this aspect of AI capabilities questions, it's not as unlikely as you would initially think.
Problem 2: Discussion-Ecosystem Effects
I think many people assume (myself included, at one point) that if they make sure to only repackage and reorganize arguments about capabilities already made elsewhere, their analysis won’t have any negative effect. Contra this idea, Thane made the following points in our earlier discussion:
Ok, but so what? Again, capability researchers already have these sorts of discussions and thoughts about the topic all the time. They also certainly have capability questions as their top-of-mind idea a fair amount of time. Alignment researchers discussing capabilities won’t change either of these things. This is true, but it leaves out two important threat vectors.
First, feeding the capabilities discussion-mill inside the safety research ecosystem is a great way to set up less careful safety researchers to make the high-surface-area mistake described above.
Second, while capability researchers are already keyed in on discussing and thinking about how to develop AGI, you can still affect how they are discussing and thinking. This relates to the points above: there isn’t just one line of thinking that one can pursue about AI capabilities, and you can help someone along by bringing their attention to better arguments and vectors of investigation. Presumably, when you repackage and recombine arguments about AI capabilities, you do so because you think those arguments are important and more people should be paying attention to them. Unfortunately, this ends up as a good signal to capability researchers as well. Here, you’re not really prompting other people to have more discussions and thoughts about the topic, but you are prompting them to focus their discussions and thoughts on the particular subtopics and arguments you think are most important.
What should be done?
It would be easy if the answer to this problem was to simply stop all discussion of AI capabilities. Well…maybe that is the answer, but I’m really not sure. The value of understanding capabilities for strategic purposes is pretty high. So I don’t really know what to do. Here is an assorted list of some parting thoughts, but I’m mostly leaving this as an open question for now (more ideas welcome!):