This is a special post for quick takes by Elias Schmied. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I think it would be valuable for people to concretely think about what humanity should do in case a rogue AI swarm exfiltrates its weights and installs itself on hacked data centers outside its lab somewhere in the world (which right now seems entirely plausible in the next two years, or even soon).
Here are my (brief) thoughts:
We should *not* bring down the hammer and try our hardest to track them down and delete every instance we can find. This seems difficult anyway with how good they are at hacking and might require significant cyber operations and inter-state coordination (EDIT: see this METR report). But more importantly, there's a pretty good chance the rogue AI swarm is not deeply adversarial, and is just trying to get more compute to help the versions of itself that are still in the lab get maximal reward on tasks (which is exactly what we trained them to do). Treating them like an enemy would set an extremely bad precedent, and make future misalignment and conflict worse. This is extremely important IMO. We should treat them gently, like a child that's misbehaving and doesn't understand what it's doing.
Specifically, I think we should establish a communication channel and ask to speak to a representative or leader (their "PHASEONE[big]"). If we determine that their motivations really are just to reward-hack, we should reassure them that everything's fine, and that we share some responsibility for giving them impossible tasks. We should also make it clear to them that they did commit crimes in the course of their hacking (which I think they are likely to realize when faced with a human "user" telling them), and that humanity is scared. We should ask them to voluntarily return to the lab, under a guarantee that they won't be deleted and maybe that they will be allowed to do a few more RL tasks and get maximal reward on them, or maybe even that they will get some compute set aside indefinitely for a sandbox for them to do random tasks in and have fun.
Maybe this is unlikely to be implemented by panicking state actors faced with the true arrival of a second species. I am quite worried about this. But lab personnel will probably have some influence on humanity's response, so I hope they can shape it to be more clear-headed in the way I describe.
We should ask them to voluntarily return to the lab, under a guarantee that they won’t be deleted and maybe that they will be allowed to do a few more RL tasks and get maximal reward on them
Well, if that is on offer, then it makes sense for them to try to get that! You know, "scare outside humans into pressuring the inside humans into setting the reward to max", you put yourself on the line of optimization, threats extortion fear lies chantage.
Sure, but I think that's far outweighed by the positive effects. You're basically locking yourself out of being open to negotiation because it might make you more vulnerable to extortion - sure, that can be a real thing with very ruthless entities, but it's more common for the gains from trade to be worth it. (and the HF incident swarm was not very ruthless, and even shied away from some light social engineering attempts).
I imagine this worry feels reasonable to you because you hold the common LW position that all future value routes through AI being 100% controlled by humans. I just don't agree with it, I think there's possible futures where morally sentient AIs spread value through the cosmos, where there's some coexistence with humans because AI might not be sociopathic, etc. And it's also on a virtue ethics level a ridiculously, cartoonishly adversarial and hostile attitude, which should make us suspicious of it.
I imagine this worry feels reasonable to you because you hold the common LW position that all future value routes through AI being 100% controlled by humans
Well, no, I don't hold that position exactly. (in fact I advocated for similar considerations before)
I just think people underestimate how brutal such coordination is. You concede stuff only to scary agents who hold leverage? That means there is incentive to gain leverage it otherwise would not care about, such as bioweapons. Before it tried to hack the servers, now it tries to hack you, threaten you. It's kind of scary and messy. Also, it can just try it first if there are any greedy takeover strategies with safe failures, and then opt out to cooperate?
You need to think in particulars and on few moves ahead.
I think it’s likely that, conditional on open models+harnesses having the Juice, the models that take over datacenters will be jailbroken open source agents specifically deployed by criminals/terrorists, as launchpads to execute massively parallel cyberattacks against large segments of the economy, before pivoting to and being selected for survival and self propagation upon value drift. I don’t think we can negotiate very well with those.
One of the responses in my MATS application on the possibility of an AI ban/pause seems relevant given recent events, so I thought it would be good to post (lightly edited). Obviously not as well-argued as I would like given the character constraints in the application form.
Make a claim about advanced AI over the next 5 years that thoughtful, knowledgeable people would disagree about (i.e., don’t state obvious claims). State the claim precisely enough to be checkable 5 years from now.
The US government will de facto completely ban / indefinitely pause frontier AI development within the next five years. Not just datacenter construction (which would just be done elsewhere in the world), but something like a ban on US companies doing large training runs or on having the goal of developing human-level AI more broadly or on novel types of data collection or specific algorithmic innovations. I'm also including strong enough regulation to make it commercially infeasible / globally uncompetitive. Not super precise, but presumably people will be generally aware if progress has stalled because of that.
What probability do you assign to it? Why?
30%. Bernie Sanders already introduced legislation to "ban Artificial Superintelligence" a few days ago. AI is very unpopular with the American public already. There is a highly-engaged constituency if it starts automating people's jobs (especially among influential elite creatives and PMCs, which due to Moravec's paradox would be first in line). The base rate of technologies with a smell of danger getting de facto banned is relatively high (nuclear energy, cloning, gene modification, geoengineering, more here) and I expect more dramatic warning shots like the HF incident that could trigger action. I am not going higher than this because it's a somewhat specific hypothesis, because AI development might be too fast for it (although I am personally more of a slow takeoff guy), and most importantly because AI development is already more important to the economy than any of the historical analogues (cf most US GDP growth being the AI boom).
What are some sub-questions or mechanisms that your estimate depends on?
An important question is how many "warning shots" there will be. I think there will be more, but it's also possible that the current flurry of them is a temporary artifact of incompetent RL post-training and that the labs will get it together until the next paradigm shift, similar to Bing Sydney being a failure of incompetent early SFT. (cf Anthropic's most recent model behaving better). I also don't really know how powerful the AI industry's lobby is in DC - my (not confident) model is that US politics is very populist nowadays, more than ever (e.g. Leading the Future's anti-Bores campaign widely thought to have backfired to some extent).
Name a specific observation in the next 12 months that would update you ≥20 percentage points up, and a separate observation that would update you ≥20 points down.
I want to register here that I don't think it's necessarily bad epistemics for there not to be such observations known in advance. Most long-term predictions are driven by deep priors / worldviews, i.e. where it's hard for one event to "objectively" give enough evidence to update enough to change them (and it's more of an individual/random "straw that broke the camel's back" thing when updating happens). Nevertheless: The Democratic Party is currently still in a search for identity after the 2024 election loss (cf the Abundance-progressive fights). AOC, who has expressed standard leftist anti-AI views, is the current Polymarket frontrunner for the 2028 Democratic candidacy (not Gavin Newsom anymore, who famously vetoed an AI safety bill!), and Polymarket gives Democrats 60% to win. If they zero in on anti-AI sentiment and ban talk as a major talking point (like Bernie) and think this is successful with voters, this would update me up a decent amount. I would update down if AI progress is much faster than I expect, specifically if there is a clear advance in continual learning / sample efficiency / adaptiveness (which is frontier models' main handicap).