tl;dr - A treaty that credibly banned “whatever weapons might end humanity” and enforced by the Japanese (or at least the Japanese national tax authority) would, I think, have a strong chance of achieving the desired objective.
Great article, thank you for all your writing. Working in Tokyo, I have what might be a useful counterexample on this take: *"This is a painfully common dynamic in corporate law. For example, multinationals routinely practice tax avoidance right up to the legal line of tax evasion. They can prove with audited books that what they’re doing is legal, even if it clearly undermines the spirit of the law."*
Experience adjacent to the Japan National Tax Authority has taught me that they are far more likely than Western tax authorities to sanction companies walking up to the line with "tax mitigation". They do this with a combination of ambiguous lines (rather than bright lines), and unilateral decisions about when a violation of the spirit of the law is an actual violation of the law, regardless of the law's text.
The downside of this approach would be potential for corruption and lack of stability. But this is entirely mitigated by (1) the NTA's professionalism and (2) the NTA's general willingness to publish Q&A explainers (with Qs posed by corporate counsel) on the NTA's approaches to tax - none of which are binding on NTA decisions if they contain loopholes exploited in bad faith, but all of which are respected and taken seriously for the effort put into them.
A hypothetical AI Authority that took its idea of authority seriously, could act in similar ways: compasionate to the concerns of AI companies in understanding the rules, but rutheless in upholding the spirit of an AGI prohibition.
Since reading this a few years ago, I've often thought about the Void and Musashi's advice. For the reference of anyone like me, the quote in its full context is comes after a description of the 'five fundamental stances' in 'the Way of the sword'. Here it is from the 2001 Wilson translation of the Five Rings -- I think it's worth thinking about together with the piece above:
The Lesson of Stance/No Stance
What is called Stance/No Stance means that there is no stance that you should take with your sword at all. However, as I place this within the Five Stances, there is a stance here. According to the chances your opponent takes, and according to his position and energy, your sword will be of a mind to cut down your opponent in fine fashion no matter where you place it. According to the moment, if you want to lower your sword a little from the Upper Stance, it will become a Middle Stance; if, according to the situation, you raise your sword a bit from the Middle Stance, it will become the Upper Stance. The Lower Stance, accordingly, may be raised a little to become the Middle Stance as well. This means that the two Side Stances, according to their position, may be moved a little to the center and become the Middle or Lower Stances.
This is the principle in which there is a stance and there is no stance. At its heart, this is first taking up the sword and cutting down your opponent, no matter what is done or how it happens. Whether you parry, slap, strike, hold back or touch your opponent's cutting sword, you must understand that all of these are opportunities to cut him down. To think, "I'll parry," or "I'll slap," or "I'll hit, hold or touch," will be insufficient for cutting him down. It is essential to think that anything at all is an opportunity to cut him down. You should investigate this thoroughly. With martial arts in the larger field, the placement of numbers of people is also a stance. All of these are opportunities to win a battle. It is wrong to be inflexible. You should make great efforts in this.
(The Japanese is here: https://www.koten.net/gorin/yaku/214/)
I had a twenty hour drive by myself recently, and binged Otherness in the Age of AGI. It was tremendous. Ambitious to take on the entire thing in one session!
Agree with this. Law is downstream of a particular medium-scale ontology of human agency. A paperclip maximizer, as mythologized, would be working with a different notion of agency, by definition.
Querying check_legality("clippy's plan"), would be like checking the temperature of the number "100". Sure, it might kinda sound like its hot, or illegal, but that's not the kind of input that check_legality() currently takes.
check_legality() can't even handle the agency of nation states ...
Genius. I'm stealing this for use in a professional setting. I'll have lawyers read chapters of "Co-Intelligence" by Ethank Mollick, and get them to focus on the useful parts for their own law firms...
(Also going to use this on "Legal Systems Very Different..." for our ACX group!)
I'm a lawyer (NY licensed) working in Tokyo, and this account of the Japanese penal system is incorrect. Prosecutors in Japan are extremely, extremely hesitant to bring a criminal case into the penal system, and so when cases are brought they are far beyond "a reasonable doubt". As a slightly misleading short summary, this, rather than lack of concern for false positives, is the reason for the notorious 99% conviction rate of criminal cases in Japan.
I've also been unhoused in a few different countries for short periods of time.
I'm certain that my affinity for Japan has its roots in needing this peaceful cultural of public safety.
Indeed, on global scale the Anti-Whatever Weapons Might End Humanity (Anti-WW MEH!) treaty would require global enforcement power to catch "off-shoring". But that's true of any global treaty!
So, is this an ex post, quite subjective decision each time? Or do they just treat all forms of tax avoidance as tax evasion?
It's a holistic decision each time taking account of everything. This is how works with almost all things we learn in law school - most obviously in criminal cases: mens rea sometimes seems to apply from sheer recklessness; an imprisonment sometimes comes out as assault & battery even when there's nothing physical about it; a killing sometimes comes out as self-defence even when its premeditated. It's only in corporate-backed actions that some legal cultures have evolved an idea that predictability is paramount and so bad faith loopholes are totally fair and reasonable to exploit until they're patched.
(Notably, multinationals are not fond of the Japanese legal system.)
(Also, notably, an overreliance on rigid black letter law was the origin of common law equity in the 14th-16th C.)