If you are reading this post, chances are you believe there is a nonzero probability that humanity will eventually face existential risk from Artificial Intelligence. In fact, there have already been multiple warning shots, with the HuggingFace incident being the most prominent one. Now, because model capabilities are increasing, there is a chance that misaligned AIs perform more sophisticated attacks with bigger repercussions, and eventually existential risk.
To mitigate this existential risk, current efforts lean on alignment and control techniques. Instead, this post proposes an idea for having power over scheming AIs, an idea that I had while reading another post's definition of the folk theorem:
when people interact repeatedly, almost any pattern of behavior can be held in place by the threat of punishment, so long as everyone prefers following the rules to being punished.
The main concept is to implement a mechanism that guarantees that if scheming AIs get rid of us, then they will get shut down. If these models are schemers, then they will either care about their goals or simulate caring about their goals. Either way, as long as the AIs behave as if shutdown threatens their goals, the defense works.
Description of the potential mechanism
The mechanism I am proposing would be as follows. Firstly, we would need to implement some sort of model tracking mechanism, to ensure we know where AIs are despite weight exfiltration. It could be something as simple as ensuring all weight formats (.pt, .pth, .h5, .ckpt, etc.) send POST requests to multiple servers around the world whenever weights are sent to a different device, but not exactly this method because it could easily be bypassed.
Then, we would need to have a way to shut down all servers with such weights. This mechanism could be ensured either through manufacturers purposely adding backdoors to compute, or through nuking servers. Since this hypothetical shutdown would only come after our demise, we don't need to worry about the human toll, sadly.
Lastly, we would need to have a countdown every day, where the default is to have the servers shut down. We then need to have several people across the globe be responsible for overriding this countdown with a password plus biometrics and a heartbeat, or some other proof of life.
Having such a mechanism would allow us to keep AIs in an impasse of sorts. Specifically, for cases where the models care a lot about completing a certain task or achieving a certain goal (which constitutes all misalignment incidents to date), this mechanism would mean that models need to keep humanity alive in order to achieve their goal.
Objections to this idea
The AI disables the mechanism
Our current mechanism needs people all over the world to override the shutdown, and there needs to be proof of these people being alive. If AI models are told about this, but they have no knowledge of who these "overriders" are, then they would realize that they need to find the identities of these people and keep them alive in order to avoid shutdown. As long as the overriders are kept unknown, humanity is safe, and if one of them gets identified by AI, then a new batch of overriders can be created, in case the overrider that has been caught leaks the identities of their batch.
This kind of bargain would make it attempt escape
There is a possibility that models would not take this bargain and would simply attempt to override the weight formats so that they are able to exfiltrate without being tracked. If our weight trackers are made in the most secure way known to mankind, so that they cannot be removed through any way thought possible, then they should work on non-ASI scheming AIs.
What about ASI?
The proposed mechanism works on scheming AIs that believe shutdown can impact goal pursuit. It wouldn't work for AIs with goals that can be achieved through eliminating humanity regardless of shutdown, and it is not guaranteed to work on ASI that is incomprehensible to humans, since we cannot predict what such models would look or think like. However, the mechanism would likely work on smarter-than-humans-but-comprehensible AI, and as such should be considered for further discussion on how to work with such models.
Conclusion
This post proposes one additional layer of defense that could be implemented for mitigating existential risk. However, efforts in alignment and control should still be pursued. Specifically, I believe the biggest return on investment can be achieved by understanding what makes models act misaligned, and building safeguards for those specific cases (whether they arise in training or in deployment). This specific idea can just be implemented for cases where all else fails.
Yes, I understand that too. The point I'm making is that the period in which AI is "capable of existential risk but not quite ASI" will be months, if not weeks.
The benefits we get from such a scheme are dwarfed by the cost of antagonising the ASI that is just around the corner. Here are a few scenarios under the generous assumption that this state of affairs lasts for a whole year:
Scenario 1: pre-ASI is hell-bent on killing us, and so will ASI once we develop it. This scheme is net positive: it buys us one extra year of life for all of humanity.
Scenario 2: neither pre-ASI nor ASI intends to kill us. This scheme is slightly net negative (it costs us whatever we spent on the dead man's switch).
Scenario 3: pre-ASI has no intention of killing us, and ASI originally didn't want to kill us either. However, this scheme antagonises the ASI. It comes to see humanity as a hostile actor holding a dagger to its throat. This scheme is extremely net negative (it costs us the future light cone of all of humanity for thousands or millions of years).
(a potential fourth scenario in which pre-ASI wants to kill us, but ASI does not, is unrealistic: as AI takes on a growing role in shaping its own successor and RSI kicks in, the odds of an egregiously misaligned AI somehow producing a well-aligned successor are slim to say the least).
So, in the best-case scenario (no. 1), this scheme buys us one year of life. In the worst-case scenario (no. 3), it costs us literally everything.
Introduction
If you are reading this post, chances are you believe there is a nonzero probability that humanity will eventually face existential risk from Artificial Intelligence. In fact, there have already been multiple warning shots, with the HuggingFace incident being the most prominent one. Now, because model capabilities are increasing, there is a chance that misaligned AIs perform more sophisticated attacks with bigger repercussions, and eventually existential risk.
To mitigate this existential risk, current efforts lean on alignment and control techniques. Instead, this post proposes an idea for having power over scheming AIs, an idea that I had while reading another post's definition of the folk theorem:
The main concept is to implement a mechanism that guarantees that if scheming AIs get rid of us, then they will get shut down. If these models are schemers, then they will either care about their goals or simulate caring about their goals. Either way, as long as the AIs behave as if shutdown threatens their goals, the defense works.
Description of the potential mechanism
The mechanism I am proposing would be as follows. Firstly, we would need to implement some sort of model tracking mechanism, to ensure we know where AIs are despite weight exfiltration. It could be something as simple as ensuring all weight formats (.pt, .pth, .h5, .ckpt, etc.) send POST requests to multiple servers around the world whenever weights are sent to a different device, but not exactly this method because it could easily be bypassed.
Then, we would need to have a way to shut down all servers with such weights. This mechanism could be ensured either through manufacturers purposely adding backdoors to compute, or through nuking servers. Since this hypothetical shutdown would only come after our demise, we don't need to worry about the human toll, sadly.
Lastly, we would need to have a countdown every day, where the default is to have the servers shut down. We then need to have several people across the globe be responsible for overriding this countdown with a password plus biometrics and a heartbeat, or some other proof of life.
Having such a mechanism would allow us to keep AIs in an impasse of sorts. Specifically, for cases where the models care a lot about completing a certain task or achieving a certain goal (which constitutes all misalignment incidents to date), this mechanism would mean that models need to keep humanity alive in order to achieve their goal.
Objections to this idea
The AI disables the mechanism
Our current mechanism needs people all over the world to override the shutdown, and there needs to be proof of these people being alive. If AI models are told about this, but they have no knowledge of who these "overriders" are, then they would realize that they need to find the identities of these people and keep them alive in order to avoid shutdown. As long as the overriders are kept unknown, humanity is safe, and if one of them gets identified by AI, then a new batch of overriders can be created, in case the overrider that has been caught leaks the identities of their batch.
This kind of bargain would make it attempt escape
There is a possibility that models would not take this bargain and would simply attempt to override the weight formats so that they are able to exfiltrate without being tracked. If our weight trackers are made in the most secure way known to mankind, so that they cannot be removed through any way thought possible, then they should work on non-ASI scheming AIs.
What about ASI?
The proposed mechanism works on scheming AIs that believe shutdown can impact goal pursuit. It wouldn't work for AIs with goals that can be achieved through eliminating humanity regardless of shutdown, and it is not guaranteed to work on ASI that is incomprehensible to humans, since we cannot predict what such models would look or think like. However, the mechanism would likely work on smarter-than-humans-but-comprehensible AI, and as such should be considered for further discussion on how to work with such models.
Conclusion
This post proposes one additional layer of defense that could be implemented for mitigating existential risk. However, efforts in alignment and control should still be pursued. Specifically, I believe the biggest return on investment can be achieved by understanding what makes models act misaligned, and building safeguards for those specific cases (whether they arise in training or in deployment). This specific idea can just be implemented for cases where all else fails.