AI scenario forecasting has a big blind spot: when AI is superhuman at hacking, what happens to the rest of society? And how does what happens to society affect AI?
To the extent that scenarios like AI 2027 address it at all, they concentrate on how cyberwarfare directly impacts the labs, such as with model weight theft. But second order effects don't get addressed.
Services whose competitors experience outages are overloaded and go down themselves.
People might stop being able to show up to work.
Organizations may decide to cut off outside network access preemptively en masse.
If organizations have issues establishing a secure outside connection at all, they may not be able to access AI-based cyber recovery services.
Presidents and prime ministers declare national emergencies and mobilize agencies to respond. However...
The workforce of cyber and disaster response teams are sized for current workloads, not for the impending massive disaster, so there would be a long queue.
Segments of the economy that ordinarily aren't thought of as bottlenecks because they're too large, might go down all at once due to correlated vulnerabilities.
Policymakers may get paranoid and start calling for war. It may become impossible to tell who or what sabotaged what.
All of this is happening while society's general coordination capability is being piecemeal degraded to pre-1950s or worse communication standards.
And so on!
And - if all this is happening - what's happening to the AI labs, exactly? Are they being drafted/nationalized as cybersecurity defenders? Are their datacenters still working? Will the datacenter buildouts go as planned? Does financing collapse? Can they afford to retain their talent? Are policymakers calling for a pause?
And we are plausibly somewhere between -1 months and 4 months before we get an open-source model+harness system, that's good enough to cause this scale of disaster, diffused into the hands of people that would like to make this happen and see the world burn.
I could go on generating examples and questions, but I hope you get my point! I think the existence of a cheap, mass-produceable superhuman coder/hacker is such a disruptive and imminent force that you really do need to factor it into your AI timelines. Apparently, the AI executives are already modelling something like this (at least, so far as it affects their own interests), but they aren't sharing yet.
And there is some adjacent prior work, asking questions like "Can AI do this?", "How bad could it get?", and "What would people do in response?", but no one is connecting this back to timelines as far as I know.
I would really like to see a public AI 2027-style simulation effort put into this. I have no experience with scenario gaming, but would give it a try if I had support and no one else was doing it. If you would like to support me doing that, please reach out!
AI scenario forecasting has a big blind spot: when AI is superhuman at hacking, what happens to the rest of society? And how does what happens to society affect AI?
To the extent that scenarios like AI 2027 address it at all, they concentrate on how cyberwarfare directly impacts the labs, such as with model weight theft. But second order effects don't get addressed.
Examples of relevant disruptive events:
And then there could be follow up effects:
And so on!
And - if all this is happening - what's happening to the AI labs, exactly? Are they being drafted/nationalized as cybersecurity defenders? Are their datacenters still working? Will the datacenter buildouts go as planned? Does financing collapse? Can they afford to retain their talent? Are policymakers calling for a pause?
And we are plausibly somewhere between -1 months and 4 months before we get an open-source model+harness system, that's good enough to cause this scale of disaster, diffused into the hands of people that would like to make this happen and see the world burn.
I could go on generating examples and questions, but I hope you get my point! I think the existence of a cheap, mass-produceable superhuman coder/hacker is such a disruptive and imminent force that you really do need to factor it into your AI timelines. Apparently, the AI executives are already modelling something like this (at least, so far as it affects their own interests), but they aren't sharing yet.
And there is some adjacent prior work, asking questions like "Can AI do this?", "How bad could it get?", and "What would people do in response?", but no one is connecting this back to timelines as far as I know.
I would really like to see a public AI 2027-style simulation effort put into this. I have no experience with scenario gaming, but would give it a try if I had support and no one else was doing it. If you would like to support me doing that, please reach out!