Much of the civilization-scale risk we are seeing in AI in 2026 comes from the following combination: we created a single institution (the "Frontier AI Company") that has two properties:

A. It is set up to create very powerful and/or self-replicating entities that may exceed the capabilities of the entirety of the rest of civilization and come with extraordinary risks

B. It gets to own an unbounded financial claim on the resulting surplus

All the technical stuff about AI, AI alignment, etc can be rolled up into point (A) above. My claim is that having point (A) on its own, without point (B) is probably okay. Nuclear technology and bioweapon technology both approximate (A) and they are mostly okay because without (B), there isn't an incentive for people controlling them to push their luck on safety.

But with Frontier AI Companies, we mixed the two.

The key claim of this post is that we can probably get rid of most of AI risk without doing anything other than separating out the bookkeeping, physical footprint and institutions so that there is no single org with both properties. And with a little help from ASICs, maybe we can also have a very productive AI industry that actually delivers most the benefits of AI to boot. No busy-waiting pause, no "banning AI", etc.

What a typical AI disaster scenario currently looks like (e.g. those by Daniel Kokotajlo):

AI company builds lots of compute and does research, with goods and services flowing in from the human economy.
Human economy mostly gets IOUs in return ("Stock")
AI company is incentivized to push forward as fast as possible so that it can sell more IOUs
AI company and/or its leading model (which is misaligned) takes over the world
IOUs are not respected, because now the misaligned superintelligence is more powerful than the mechanisms that were supposed to enforce the IOUs
Everyone dies, The End

Obviously, the humans made a mistake in letting the AI company write the IOUs to them. But once there's a system for stocks, banking, money, etc, letting people write you IOUs is the default and it's hard to stop.

So, here's how I would separate the two. Create two new kinds of entity:

(1) - AI research and development (AI R&D) organizations.

(2) - AI deployment companies

AI R&D orgs would do everything that involves training, distilling, testing, and aligning frontier AI models. They would be heavily legally restricte...

Steve Scott

Steve Scott