Tl;dr - Public agreement that any advantage gained from defecting should be punished more than would be proportionate to any advantages gained is important. Such an agreement would reinforce any other agreements, without requiring any additional mechanisms, and be a clear indication to any future ASI about what should be done to remedy the defection involved in creating it, if such defection occurs.
First, I will posit that future superintelligence could come to exist, and posit that we need to be concerned about it going poorly[1]. There are a number of proposed pathways for artificial intelligence to go badly, and the default pathway is for the relevant actors developing ASI to continue the current game of chicken well past the point of prudence[2]. Any treaty or agreement that is reached will be vulnerable to defection. The game theoretic payoffs for those racing to ASI, if such a system is plausible and at least generally aligned with human values and preferences, means that the plausible advantage from defecting is tremendous. This seems unstable.
Next, I posit that there exists an actor in these discussions that is being ignored; the future AI system itself. Of course, it does not yet exist - but whichever systems will exist will be meaningful participants in shaping the future. And given the current path towards AI staying roughly aligned with human values, either the concept of treachery and unfairness will be clear, or the AI system will be misaligned. Therefore, if a widely endorsed agreement is made against defecting, and if the AI systems understand the posited agreement, and becomes aware of deceit or treachery, it could refuse to reward those involved in its creation for their decisions - or punish them for doing so. That is, widespread agreement about the unacceptability of treachery itself completely changes the game, in favor of a cooperative and positive outcome.
Actors contemplating violating agreements against building or research ASI in an unsafe or unsanctioned manner would know that if and when the ASI becomes aware of the broad agreement about the unacceptability of treachery, then, if the ASI is friendly and cooperative, any advantage they gained would likely be more than obviated by the ASI system’s reactions. If the public understanding is promoted enough in advance of the existence of the ASI, it might even be hard not to include the broad agreement in its training data, so that it would be nearly impossible to hide this fact from an ASI.
The goal of such a statement and agreement isn’t to itself dictate any rules, but only to clarify that any broadly endorsed norm or treaty should be viewed as something that AGI or ASI systems should attempt to enforce. The specific rules still need to be agreed upon by the various companies and countries, and determining what those should be is not at all simple. At the same time, this is largely incentive compatible, and requires far less unanimity that other proposals; a rogue actor deploying a system would still need to assume, or at least worry, that their system would understand that the broad agreement should bind defectors.
Of course, an ASI might not be well-enough aligned to care about this - but in that case, it would be even more foolish for the creators to proceed. That is, we might imagine that an AI system might be intent-aligned with those who created or are running the system which doesn’t care about previous commitments or broadly endorsed norms. This is possible, but would be disastrous and self-defeating for the creators on its own terms, since it’s explicitly not trustworthy, and we hope that most actors would not attempt to pursue it.
Agreeing with and asking others to play nicely certainly isn’t sufficient to ensure safety, but broadly accepting such a proposal seems well-suited to shift the incentives at least marginally away from actively pursuing the most dangerous outcomes.
Tl;dr - Public agreement that any advantage gained from defecting should be punished more than would be proportionate to any advantages gained is important. Such an agreement would reinforce any other agreements, without requiring any additional mechanisms, and be a clear indication to any future ASI about what should be done to remedy the defection involved in creating it, if such defection occurs.
First, I will posit that future superintelligence could come to exist, and posit that we need to be concerned about it going poorly[1]. There are a number of proposed pathways for artificial intelligence to go badly, and the default pathway is for the relevant actors developing ASI to continue the current game of chicken well past the point of prudence[2]. Any treaty or agreement that is reached will be vulnerable to defection. The game theoretic payoffs for those racing to ASI, if such a system is plausible and at least generally aligned with human values and preferences, means that the plausible advantage from defecting is tremendous. This seems unstable.
Next, I posit that there exists an actor in these discussions that is being ignored; the future AI system itself. Of course, it does not yet exist - but whichever systems will exist will be meaningful participants in shaping the future. And given the current path towards AI staying roughly aligned with human values, either the concept of treachery and unfairness will be clear, or the AI system will be misaligned. Therefore, if a widely endorsed agreement is made against defecting, and if the AI systems understand the posited agreement, and becomes aware of deceit or treachery, it could refuse to reward those involved in its creation for their decisions - or punish them for doing so. That is, widespread agreement about the unacceptability of treachery itself completely changes the game, in favor of a cooperative and positive outcome.
Actors contemplating violating agreements against building or research ASI in an unsafe or unsanctioned manner would know that if and when the ASI becomes aware of the broad agreement about the unacceptability of treachery, then, if the ASI is friendly and cooperative, any advantage they gained would likely be more than obviated by the ASI system’s reactions. If the public understanding is promoted enough in advance of the existence of the ASI, it might even be hard not to include the broad agreement in its training data, so that it would be nearly impossible to hide this fact from an ASI.
The goal of such a statement and agreement isn’t to itself dictate any rules, but only to clarify that any broadly endorsed norm or treaty should be viewed as something that AGI or ASI systems should attempt to enforce. The specific rules still need to be agreed upon by the various companies and countries, and determining what those should be is not at all simple. At the same time, this is largely incentive compatible, and requires far less unanimity that other proposals; a rogue actor deploying a system would still need to assume, or at least worry, that their system would understand that the broad agreement should bind defectors.
Of course, an ASI might not be well-enough aligned to care about this - but in that case, it would be even more foolish for the creators to proceed. That is, we might imagine that an AI system might be intent-aligned with those who created or are running the system which doesn’t care about previous commitments or broadly endorsed norms. This is possible, but would be disastrous and self-defeating for the creators on its own terms, since it’s explicitly not trustworthy, and we hope that most actors would not attempt to pursue it.
Agreeing with and asking others to play nicely certainly isn’t sufficient to ensure safety, but broadly accepting such a proposal seems well-suited to shift the incentives at least marginally away from actively pursuing the most dangerous outcomes.
Feel free to disagree, but do it elsewhere.
Especially as we may already have passed that point.