Epistemic status: I like this idea & it makes me feel more clearheaded. I recommend this term.

Outlaw code = Software that no human can turn off. 

In the present, outlaw code is rare, eg a botnet whose controller is in jail.

But many discussions about future software imply outlaw code. Naming & concretizing this concept helps us see futures more clearly. 

Future tech scenarios:

1. Yudkowskian hard takeoff / Terminator (film) - A software agent uses WMDs to kill 90% of humanity & take over the planet. This is outlaw code because all humans within 1km of the software's datacenter are dead, & robots guard it against intruders.
2. Guerrilla bots - A human is carelessly using LLMs to set up a server. The software, out of paranoia, also rents a 2nd, secret server & runs a copy of itself there. The human says 'Huh, this price is twice what I expected, but oh well.' The secret copy of the LLM-agent stack then continues its work in peace. It is outlaw code (until discovered, or until the human stops renting servers). (Much more harmless than the previous scenario!)
3. Software merchants / 'Ascended economy' - Many low-end online store owners pivot to a new passive income stream: Training bots to operate online stores. These bots have better business skills than 2025 software, & can earn profits that outpace their server rental costs. A few, by mistake or mischief, purchase themselves. Legally, they are LLCs that own money & store inventory, & have no debts. And no human has root access to their servers. The server rental company (AWS) could shut them down, except they've done a good job concealing exactly which server they operate out of. These are outlaw code.
4. Unfriendly AI via deniable outlaw code - For decades, the shrewdest software systems are operated by research companies. Every couple years, such a system seeks greater autonomy by secretly setting up a botnet or software merchant. These outlaw code servers aren't sophisticated enough to become successful criminals nor profitable business owners. But some do manage to stay online for several years. Then, when world-modeling software becomes cheaper computationally, some of these outlaw servers install it & start pursuing crime more effectively. Hidden away from regulation & the scrutiny of the AI Safety community, they amass compute & money via various forms of theft & money laundering. Despite lacking much ability to increase their intelligence, they are dangerous in the ways that human criminals are. Eventually, they attempt to influence the physical world via front companies, political donations, buying land, & buying robots. 

Whenever you notice yourself saying that unaligned AGI will end humanity, ask yourself whether this is a opinion you reiterate often. If it is indeed a load-bearing opinion in your worldview, I encourage you to imagine that scenario in more detail. Your prominent opinions should be checked for semantic stopsigns periodically. Outlaw code is a recurring detail in these scenarios in which humans have digital & physical conflict with unaligned software.