In this post, I propose adapting banking risk management frameworks (specifically capital adequacy requirements like Basel III) to frontier AI labs. By forcing them to hold capital reserved proportionate to their model misalignment risks, we align market incentives directly with catastrophic risk mitigation. In so doing, this would give frontier AI labs' alignment researchers an incentive structure with lower levels of moral hazard.

I write this post from my perspective as a former investment banking macro trader, researcher, now working in Explainable AI.

Frontier AI - long with no risk-management oversight

In banks (and to a less stringent degree, hedge funds), traders/PMs operate under the oversight of risk management teams. Post 2008, risk management teams got beefed up, with policymakers passing laws forcing banks to give them more say in how a trading desk operates. The introduction of laws, such as Basle II/III (capital adequacy) and the UK’s Senior Management Regime, put much great personal accountability on senior management in banks for the risk that their traders were taking. That gave banks the incentive to add more risk oversight to the operations. Under Basle, banks had to hold capital against their risk-weighted assets - get long risk, place capital at the central bank in case it goes wrong.

Now that I am no longer trading, instead focusing on Explainable AI research and AI alignment, I see a the race to the moon of AI Frontier black-box labs, and the nascent AI Alignment movement (and eventual industry) as analogous to the trading/risk-management relationship.

Today’s frontier AI labs mirror pre-2008 trading desks:

• Growing leverage: Enormous balance sheets engaged in circular investment and infrastructure deals.
• Moral hazard: Labs have hired in-house alignment researchers, but compensate them in cash and stock of the lab they monitor.
• Missing independent risk limits: Current voluntary commitments and Responsible Scaling Policies (RSPs) lack external enforcement mechanisms with financial teeth.

The analogy of frontier labs to LTCM comes to me a lot (a multi-leg, multi-counterparty repo-funded money-printing machine until it wasn’t). 

The Frontier labs have hired their own Alignment researchers but, given that their Alignment researchers are compensated by cash and stock of the Lab, their alignment is not necessarily aligned to Alignment. In-house Alignment researchers therefore (currently) present a moral hazard to the AI industry.

Internalising Misalignment Risk (ex-ante capital vs. ex-post liability

Prior governance discussions on LessWrong have explored strict tort liability regimes, mandatory liability insurance for catastrophic risk, and private insurance as a regulatory pathway.

While insurance and tort liability primarily target ex-post compensation (after a loss event occurs), a capital adequacy framework targets ex-ante balance sheet liquidity:

1. Mandatory Reserves: Regulators or auditing standards bodies require frontier labs to set aside liquid capital reserves proportional to their evaluated misalignment risk score.
2. Dynamic Incentives: As a model's risk score increases, its capital requirement rises, directly reducing the capital available for R&D, compute acquisition, or equity buybacks.
3. Empowering In-House Alignment: Alignment researchers gain organisational power analogous to bank risk officers. Demonstrating higher safety and verifiability directly frees up balance sheet capital for the firm. Alignment gets rewarded in the market.

An insurance-based, ex-post mechanism would, in my opinion, be less financially efficient than a capital-adequacy framework. It would require a derivatives market to be created for it to adjust premiums proactively but, for that to be liquid in the market there would need to be active two-way demand (insurer wants to buy misalignment protection, but who wants to sell it?). I may easily be missing half of the picture here and would welcome discussion on this.

Pricing Misalignment into Market Forwards

Markets price potential balance-sheet drags into forward earnings valuations. This is how a capital adequacy alignment regime would get enforced. The labs would maybe have Alignment as a board seat. This therefore impacts the equity of the AI labs (once listed) and, once mature, their credit markets too. It would probably play out via SpaceX, Meta, Google etc. currently.

To illustrate how it might impact market pricing, consider what happens as AI capabilities scale exponentially. The capital buffer required for a high-risk, black-box model will grow faster than raw token revenue can offset. It would only not do this for a lab which was scaling a perfectly aligned model.

Misalignment risk would likely materialise as a direct drag on forward earning ratios. For an unlisted lab, forward revenue multiples for future capital raising would likely be lower.

Once markets discount a negative economic consequence for misaligned frontier models, the economic incentive to solve alignment becomes embedded directly into market dynamics.

Metrics for Evaluating Misalignment

To calculate a lab’s Risk-Weighted Capital Reserve will require solid metrics. The metrics need to be robust to gaming, credible and likely produced, checked (and subject to ongoing review and improvement) by Independent Alignment researchers.

Coming from a background in neurosymbolic and explainable AI, one promising direction involves measuring deviations from provable outputs or formal constraints. A live research area in NSAI is to understand how much of a frontier lab's LLM's output can be proven correct/incorrect.

In the near term formal verification alone does not equal "alignment". A realistic framework must combine formal safety proofs with empirical red-teaming, behavioral evaluation suites, and architectural transparency.

Indeed, if the Frontier labs continue to drive opaque architectures and closed-source/closed-weights (these become less desirable operational models if RWCR is implemented), MI and other disciplines will likely be leading the way in estimating misalignment, with Formal Methods following closely.

An Example Risk-Weighting Framework (Illustrative)

To illustrate our thinking, both in terms of the different methods needed to measure alignment, and the economic cost needed to applied to incentivise alignment, we have constructed a table. By considering model capability tier on one axis, this would allow an entire lab (or a corporate using AI in their own operations) to calculate a Risk-Weighted Token (RWT) reserve ratio:

A lab deploying an unverified Tier 3 model would face a steep reserve requirement, especially if that model got widespread adoption. Companies creating narrow, accurate models would be liable only for a small reserve. This should create financial incentive to down-tier risk through verifiable safety architectures before scaling deployment. It would likely mean that frontier labs would take longer to assess and refine their latest models. It wouldn't stop the "arms race", but would apply a significant degree of caution to it.

For corporate consumers of AI models, we would expect that misalignment risks would also feature in their business models and therefore the market pricing of their equity and credit risk. This would complete the circle, incentivising a publicly listed corporate to use AI which met its own needs for Intelligence and Alignment.

Open Questions for Discussion

1. Compute vs. Revenue Baseline: Should capital adequacy be calculated against annual token revenue, total compute expenditure (capex), or deployed FLOPs?
2. Regulatory Capture: How do we prevent established frontier labs from lobbying for reserve rules that serve as a regulatory moat against open-source or smaller entrants?
3. Audit Standardisation: What independent bodies are best equipped to audit risk-weighted reserves without relying on self-policing by lab-funded researchers?