We recently (finally!) got the results of the 2024 survey out. The paper is here, but it’s pretty long, so I’ll tell you the most interesting bits (according to me).
But first, quick background: this was the fourth run of the same survey since 2016. We wrote to everyone we could who published in six top-tier AI venues and got a 10% response rate—high! We got 1580 valid responses, but don’t be confused: specific questions often have answers from many fewer researchers, because we gave each person a randomized subset (see Section 2.5). There was almost certainly some non-response bias, but it probably doesn’t make much difference. Researchers filled out the survey in December 2024, so some things have probably changed.
To me the most striking results are about extinction or disempowerment (Section 3.9). On average researchers put an 18% chance on “future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species”. Over half said at least 10% and one in three said at least 20%. From the comments, I think people are thinking of a variety of extreme disempowerment scenarios here, not just extinction.
And Zooming in, another really interesting thing is that researchers educated in Asia had higher extinction/disempowerment numbers than US or European researchers! This is interesting because a common defense of pushing forward with dangerous AI is that the US is in an arms race with China, and (implicitly) China won’t want to cooperate.
This actually looks even stronger for researchers taking the survey from China, but we aren’t sure about this result yet, so it’s not in the paper—we are digging into it because the number of researchers looks too small, and we want to know how much that’s down to VPNs, a lower response rate, or something else.
(Preliminary results, work in progress)
Looking beyond extinction and disempowerment, researchers saw a lot of concerning scenarios on the horizon. Here’s a map of how much concern different researchers thought different concern deserved—it’s notable to me that it is so full of orange and red:
I find this partly striking because the participants are AI researchers, who I might expect to be relatively positive on their field. Also interesting to me: the scenarios that seem most likely to lead to extinction out of this set are not clustered at the top.
We asked about a bunch of traits relevant to future AI having the capacity to behave dangerously. Interestingly to me, in the year since the previous survey they systematically shifted: more people expected nearly all of them twenty years out (and that’s only good news for one of them).
Researchers were quite pessimistic about being able to understand AI decisions by 2029. Here’s “For typical state-of-the-art AI systems in 2029, do you think it will be possible for users to know the true reasons for systems making a particular choice?…”
This is normal, but remains interesting in the context of moves to hand swathes of decisions to AI. I think it might also be striking to a lot of people who don’t understand that the people creating AI systems do not program them, and have very little idea what is going on inside them. If this question was about traditional software, answers would look very different.
Overall, if we build AI that does task humans can do, researchers see an extremely uncertain future with a lot of bad. I love the below graph, which lines up every participant in order of pessimism and shows their opinion on how likely different outcomes are as a single vertical bar. But I have also never seen so many people so passionately hate a single graph as when we posted a version of this on Twitter in an earlier survey iteration, so feel free to skip!
I always find the results on how much different inputs to research matter interesting. We are asking people how much less total AI progress would have happened if different inputs had been halved over the past decade. And the average is pretty near half for all of them, whereas I might have thought e.g. loss of hardware progress would have hit harder than some of the others. But also, views are so spread out! Some people think losing half of dataset effort would have destroyed progress while others think it would barely matter, and this is close to true for each input.
We asked when AI would be able to do a bunch of narrow tasks, such as independently fine-tune an open-source LLM or write what sounds like a new Taylor Swift song to a fan. Timelines for these were short: all but five within ten years in the average forecast, and all but one had 10% by 2027. (The remaining one was independently solving something like a Millennium Prize problem—which hasn’t been conclusively achieved because the recent solution wasn’t without human involvement, but I bet this would get shorter timelines now.)
We asked when AI would surpass human-level performance via two different kinds of questions, one focused on tasks and one on occupations. Researchers always give extremely different answers to these, mysteriously to me. So we have one measure that puts human-level AI 16 years from now (as likely as not, in the average distribution), and one that puts it in 72 years. More interestingly to me, these timelines are falling very fast. In the year or so since the previous survey, dates for broadly human-level AI dropped five years in the task-based question and seventeen-years in the occupation-based question. Also, for the task-based question, the average forecast put 10% on this by 2027!
In 2024, people were split on what speed of global AI research would make them most optimistic—about a third each way between faster, slower, and the current speed. This seems like a place I expect to be most out of date! I’ll be curious to see the 2026 results, which we hope to get out to you in 20261.
We recently (finally!) got the results of the 2024 survey out. The paper is here, but it’s pretty long, so I’ll tell you the most interesting bits (according to me).
But first, quick background: this was the fourth run of the same survey since 2016. We wrote to everyone we could who published in six top-tier AI venues and got a 10% response rate—high! We got 1580 valid responses, but don’t be confused: specific questions often have answers from many fewer researchers, because we gave each person a randomized subset (see Section 2.5). There was almost certainly some non-response bias, but it probably doesn’t make much difference. Researchers filled out the survey in December 2024, so some things have probably changed.
To me the most striking results are about extinction or disempowerment (Section 3.9). On average researchers put an 18% chance on “future AI advances causing human extinction or similarly permanent and severe disempowerment of the human species”. Over half said at least 10% and one in three said at least 20%. From the comments, I think people are thinking of a variety of extreme disempowerment scenarios here, not just extinction.
And Zooming in, another really interesting thing is that researchers educated in Asia had higher extinction/disempowerment numbers than US or European researchers! This is interesting because a common defense of pushing forward with dangerous AI is that the US is in an arms race with China, and (implicitly) China won’t want to cooperate.
This actually looks even stronger for researchers taking the survey from China, but we aren’t sure about this result yet, so it’s not in the paper—we are digging into it because the number of researchers looks too small, and we want to know how much that’s down to VPNs, a lower response rate, or something else.
Looking beyond extinction and disempowerment, researchers saw a lot of concerning scenarios on the horizon. Here’s a map of how much concern different researchers thought different concern deserved—it’s notable to me that it is so full of orange and red:
I find this partly striking because the participants are AI researchers, who I might expect to be relatively positive on their field. Also interesting to me: the scenarios that seem most likely to lead to extinction out of this set are not clustered at the top.
We asked about a bunch of traits relevant to future AI having the capacity to behave dangerously. Interestingly to me, in the year since the previous survey they systematically shifted: more people expected nearly all of them twenty years out (and that’s only good news for one of them).
Researchers were quite pessimistic about being able to understand AI decisions by 2029. Here’s “For typical state-of-the-art AI systems in 2029, do you think it will be possible for users to know the true reasons for systems making a particular choice?…”
This is normal, but remains interesting in the context of moves to hand swathes of decisions to AI. I think it might also be striking to a lot of people who don’t understand that the people creating AI systems do not program them, and have very little idea what is going on inside them. If this question was about traditional software, answers would look very different.
Overall, if we build AI that does task humans can do, researchers see an extremely uncertain future with a lot of bad. I love the below graph, which lines up every participant in order of pessimism and shows their opinion on how likely different outcomes are as a single vertical bar. But I have also never seen so many people so passionately hate a single graph as when we posted a version of this on Twitter in an earlier survey iteration, so feel free to skip!
I always find the results on how much different inputs to research matter interesting. We are asking people how much less total AI progress would have happened if different inputs had been halved over the past decade. And the average is pretty near half for all of them, whereas I might have thought e.g. loss of hardware progress would have hit harder than some of the others. But also, views are so spread out! Some people think losing half of dataset effort would have destroyed progress while others think it would barely matter, and this is close to true for each input.
We asked when AI would be able to do a bunch of narrow tasks, such as independently fine-tune an open-source LLM or write what sounds like a new Taylor Swift song to a fan. Timelines for these were short: all but five within ten years in the average forecast, and all but one had 10% by 2027. (The remaining one was independently solving something like a Millennium Prize problem—which hasn’t been conclusively achieved because the recent solution wasn’t without human involvement, but I bet this would get shorter timelines now.)
We asked when AI would surpass human-level performance via two different kinds of questions, one focused on tasks and one on occupations. Researchers always give extremely different answers to these, mysteriously to me. So we have one measure that puts human-level AI 16 years from now (as likely as not, in the average distribution), and one that puts it in 72 years. More interestingly to me, these timelines are falling very fast. In the year or so since the previous survey, dates for broadly human-level AI dropped five years in the task-based question and seventeen-years in the occupation-based question. Also, for the task-based question, the average forecast put 10% on this by 2027!
In 2024, people were split on what speed of global AI research would make them most optimistic—about a third each way between faster, slower, and the current speed. This seems like a place I expect to be most out of date! I’ll be curious to see the 2026 results, which we hope to get out to you in 20261.
Relatedly, if you know someone who would be great at overseeing surveys like this, please send them our way (caitlin@aiimpacts.org).