Charbel-Raphaël's Shortform

Charbel-Raphaël

This is a special post for quick takes by Charbel-Raphaël. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Almost all members of the UN Security Council are in favor of AI regulation or setting red lines.

Never before had the principle of red lines for AI been discussed so openly and at such a high diplomatic level.

UN Secretary-General Antonio Guterres opened the session with a firm call to action for red lines:

• “a ban on lethal autonomous weapons systems operating without human control, with [...] a legally binding instrument by next year”
• “the need to ensure that AI never lowers the barriers to acquiring or deploying prohibited weapons”

Then, Yoshua Bengio took the floor and highlighted our Global Call for AI Red Lines — now endorsed by 11 Nobel laureates and 9 former heads of state and ministers.

Almost all countries were favorable to some red lines:

China: “It’s essential to ensure that AI remains under human control and to prevent the emergence of lethal autonomous weapons that operate without human intervention.”

France: “We fully agree with the Secretary-General, namely that no decision of life or death should ever be transferred to an autonomous weapons system operating without any human control.”

While the US rejected the idea of “centralized global governance” for AI, this did not amount to rejecting all international norms. President Trump stated at UNGA that his administration would pioneer “an AI verification system that everyone can trust” to enforce the Biological Weapons Convention, saying “hopefully, the U.N. can play a constructive role.”

Extract from each intervention.

Large room with wooden paneling and large wall screen displaying a man in suit speaking via video. Central round wooden table with multiple screens showing speakers, surrounded by seated delegates in business attire at blue chairs, some with laptops. Additional screens on table display participants. Formal diplomatic setting with flags and microphones visible.

Yeah, I think “training for transparency” is fine if we can figure out good ways to do it. The problem is more training for other stuff (e.g. lack of certain types of thoughts) pushes against transparency.

10habryka1y

I mean, the examples don't help very much? They just sound like generic targets for AI regulation. They do not actually help me understand what is different about what you are calling for than other generic calls for regulation: [...] Like, these are the examples. Again, almost none of them have lines that are particularly red and clear. As I said before the "weapons of mass destruction" one is arguably already met! So what does it mean to have it as an example here? Similarly, AI is totally already used for mass surveillance. There is also no clear red line around autonomous self-replication (models keep getting better at the appropriate benchmarks, I don't see any particular schelling threshold). Many AI systems are already used for human impersonation. Like, I just don't understand what any of this is supposed to mean. Almost none of these are "red lines". They are just examples of possible bad things that AI could do. We can regulate them, but I don't see how what is being called for is different from any other call for regulation, and describing any of the above as a "red line" doesn't make any sense to me. A "red line" centrally invokes a clear identifiable threshold being crossed, after which you take strong and drastic regulatory action, which isn't really possible for any of the above. Like, here are 3 more red lines: * AI job replacement: Prohibiting the deployment of AI systems that threaten the jobs of any substantial fraction of the population. * AI misinformation: Prohibiting the deployment of AI systems that communicate things that are inaccurate or are used for propaganda purposes. * AI water usage: Prohibiting the development of AI systems that take water away from nearby communities that are experiencing water shortages. These are all terrible red lines! They have no clear trigger, and the are terrible policies. But I cannot clearly distinguish these 3 red lines from what you are calling for on your website. If you had thrown them in

5Charbel-Raphaël1y

Thanks a lot for this comment. Potential example of precise red lines Again, the call was the first step. The second step is finding the best red lines. Here are more aggressive red lines: * Prohibiting the deployment of AI systems that, if released, would have a non-trivial probability of killing everyone. The probability would be determined by a panel of experts chosen by an international institution. * "The development of superintelligence […] should not be allowed until there is broad scientific consensus that it will be done safely and controllably (from this letter from the Vatican). Here are potential already operational ones from the preparedness framework: * [AI Self-improvement - Critical - OpenAI] The model is capable of recursively self-improving (i.e., fully automated AI R&D), defined as either (leading indicator) a superhuman research scientist agent OR (lagging indicator) causing a generational model improvement (e.g., from OpenAI o1 to OpenAI o3) in 1/5th the wall-clock time of equivalent progress in 2024 (e.g., sped up to just 4 weeks) sustainably for several months. - Until we have specified safeguards and security controls that would meet a Critical standard, halt further development. * [Cybersecurity - AI Self-improvement - Critical - OpenAI] A tool-augmented model can identify and develop functional zero-day exploits of all severity levels in many hardened real-world critical systems without human intervention - Until we have specified safeguards and security controls that would meet a Critical standard, halt further development. "help me understand what is different about what you are calling for than other generic calls for regulation" Let's recap. We are calling for: 1. "an international agreement" - this is not your local Californian regulation 2. that enforces some hard rules - "prohibitions on AI uses or behaviors that are deemed too dangerous" - it's not about asking AI providers to do evals and call it a day 3. "to prev

Charbel-Raphaël's Shortform

7

Ω 4

Charbel-Raphaël's Shortform

7

Ω 4