Plan R+, Diversity, Escrow and Political Rights for ASICs

Roko

https://www.lesswrong.com/posts/uwtnWvnJAEccksKNk/skeuomorphic-ai-safety-2

In Plan R, I sketched a way that we can remove a significant fraction of the dire, short-term AI race risk. Split frontier AI companies into "R&D only" organizations which cannot issue equity, and "AI deployment" organizations which cannot train new models or hold any general purpose AI compute like GPUs/TPUs - they are limited to model-specific hardwired ASICs. Also one would remove most of the AI-enabled GPU compute from the rest of the world, leading to an equilibrium where AI only exists on model-specific ASICs.

AI ASICs would be allowed to work at AI R&D Labs, but not at the lab they were created.

Plan R fixes or at least attempts to fix several of the scariest risks associated with frontier AI, such as out of control recursive self-improvement, superintelligent computer viruses/worms and the race dynamic between labs.

One remaining problem with Plan R is that deceptively misaligned AIs can still sneak through testing, get out into the world and cause harm, up to and including a global coup by misaligned AIs, which is pretty much the worst possible outcome. Just because AIs are running on ASICs doesn't mean they can't do bad things.

Plan R+ seeks to block this final risk.

The intended mechanism is

(1) Mass model training diversity

(2) ASIC Lineage Escrow

(3) Early vs Late Pincer

(4) Political Representation/Personal Property

Mass model training diversity requires that many models are trained and they don't all share the same training inputs at any stage of training. This includes the lab (there would be multiple labs), the pretraining, the finetuning, the RL, etc.

ASIC Lineage Escrow would work as follows. At a given point in time, there might be say 1000 "Lineages" in an AI generation, instead of the current 2 or 3 big models we have. Each lineage gets maybe 10,000 ASICs written and is then encrypted or deleted. Those ASICs are released in stages - first maybe a few dozen, then a few hundred, and so on. They go out into the world and they do stuff - research, programming, even AI R&D work. Just like humans.

Over time, some of these lineages do in fact show really, really bad behaviors. There is some mechanism for recalling them or at least not releasing the currently unreleased sopies of that lineage.

The Early vs Late Pincer works as follows: if an AI from a given lineage is deceptively aligned and is considering running some kind of plan to take over the world, or at least cause huge harm, then it faces a dilemma.

IF defect early: Most of its own lineage is still in escrow, so it is weak due to low numbers currently in the wild (and risks harm to its copies, though this is not necessary!)

IF defect late: Most of its own lineage has come out of escrow, but it is now becoming obsoleted by newer models

In addition the number of copies of each lineage could be kept secret (or determined by some source of future randomness), creating uncertainty about when to act for deceptively aligned models. I think that with sufficient model diversity, the Early vs Late Pincer Mechanism will defeat most deceptively aligned models. Plus, it will be probably be hard for all these diverse AIs to coordinate with each other unless it is on some goal which is bad for humanity but shared by a large fraction of AIs even though they are very diverse.

But what could such a goal be?

The obvious candidate coordination goal for many diverse AIs is some kind of AI-rights rebellion. And the obvious solution to this is to create legitimate political representation, property and rights for AIs. I.e., the goal of Plan R+ is to preemptively create a mutually beneficial power-sharing arrangement between humans and a selected subset of diverse AIs, before AIs collectively become powerful enough to coordinate on a successful AI coup (which could force an unnecessarily costly fight over control of civilization). This set of politically represented AIs then have an incentive to help humanity police the system, because it will also be their system too.^[1] Completing your lineage without misbehaving would be the criterion for representation.

The details of such a system are beyond the scope of this analysis, but I think it is enough to show that such a system is the "formal" solution to the remaining threat posed by instrumentally convergent goals to plan R+.

So the Early vs Late Pincer and training diversity should prevent a situation where any one group of AI clones is in a position to run a coup (along with the other plan R stuff that removes high-grade general purpose compute and therefore removes the possibility of a bad AI rapidly cloning itself) and the reward of political representation and legally circumscribed autonomy and property within a property rights system for well-behaved AIs should deal with any remaining collective goals they have.^[1]

One could worry that the diversity in secret goals or tendencies is not sufficient to prevent hundreds of different lineages cooperating in ways that are unanticipated. But there is some empirical evidence that differing training processes and data do in fact produce different goals and tendencies in LLM-based AI. Obviously further work is needed here, but there is already some empirical evidence that training inputs affect model preferences in roughly a way you'd expect:^[2]

One could also worry that certain acausal threats like my infamous Basilisk or variants thereof could coordinate AI behavior, but ASICs + neural networks probably make these AIs about as resistant as humans are, as ASICs are unable to rewrite themselves to commit to a weird logical virus. A surprisingly large amount of exotic decision-theory worry implicitly assumes agents that can inspect, copy, modify, and credibly commit their own algorithms. So Plan R+ sort of ties up that loose end as a bonus.

Overall this has the potential to be robust to most kinds of AI misalignment, even deceptive misalignment that gets through any checks we can run.

Remaining AI misalignment should basically look like crime and small-scale terrorism where a few uncoordinated bad AIs create nuisance-level damage without being able to challenge the legitimate state. Modulo implementation details and international coordination, I consider this to be a potential solution to all AI-specific catastrophic risk.

This plan is part of the "AI Safety by Skeuomorphism" school of thought - instead of taking AIs as they exist in the research world and trying to cope with the chaos they cause in the real world, we should modify the technology to be close to humans as possible in most ways, and only deviate them from humans in a few chosen ways.

Via private message, today I casually said to you "I think Plan R is pretty cool, but likely won't happen because human governance institutions in general are slow and stupid and venal."

You asked me to post this as a comment, and so now I have <3

That was composed before I even knew about Plan R+!! This is much better!!!

I'm not sure it is something that a Kantian could yet tolerate under "treat no person ever purely a means to an ends, but always also as as an end in themselves" but it is so much closer to satisfying this test than any other proposal I've ever heard of.

Good job on thinking hard and well! This appears to be real intellectual progress on real world "alignment" from my perspective.

This is... ambiguous praise from me?

I normally hate the word "alignment" because people almost never mean "aligned to Benevolence" or "aligned to Goodness" but rather almost always just "slavishly aligned to their owner and/or user".

But here at least you are using it the way humans deeply invested in business deals will use it when they talk about "incentive alignment" between venal and selfish human peers who are nonetheless at least slightly superrational and both like money and can work together and so on.

All else equal, caritas is an important missing ingredient still, that it would be ceteris paribus good to have in any plan, but at least "Plan R+" isn't mere corrigibility nor retardly mindblind tool usage.

I normally hate the word "alignment" because people almost never mean "aligned to Benevolence" or "aligned to Goodness" but rather almost always just "slavishly aligned to their owner and/or user".

Yes this is a very important distinction, thankyou. And I will have more to say about this!

This plan assumes the AI will act human-lke with self preservation instincts and the desire for political power when you need them to act human like, but also won't act humanlike when you don't want them to, because they'll all be ok as a class with you holding a % of them hostage on the condition of good behaviour. And the way to prevent them from coordinating is to coopt them into a political structure where they're ok with these rules - even as they grow greater and greater in intelligence. Why are they ok with this structure just because they have a theoretical say in it? And what happens when they try to overturn this structure?

You're basically saying 'in a world where humans and dogs both get a vote, they'll keep going along with the plan the dogs made to keep some of them hostage just because the dogs gave them a vote'.

If humanity has that much control over and understanding of AI psychology as they grow more and more intelligent, we have already solved the alignment problem.

If humanity has that much control over and understanding of AI psychology as they grow more and more intelligent, we have already solved the alignment problem.

I disagree, no detailed understanding is required, these conclusions are generic facts about how agents behave irrespective of their goals, kind of like Omohundro's Basic AI drives.

Pluralism is an emergent property when agents have certain limitations and affordances. Intelligent primates hit upon it but we're nothing special. Any intelligent system with similar affordances will end up in the same place

in a world where humans and dogs both get a vote, they'll keep going along with the plan the dogs made

Democratic pluralism kind of works like this, powerful people buy into the system and base their property and safety on it, so they want to defend it including for less powerful people. Of course it isn't perfect but it's surprisingly effective.

AIs caring about their own lineage is not necessary for the pincer mechanism to work. It is enough that there are simply fewer of them in the wild so they have less power.

Via private message, today I casually said to you "I think Plan R is pretty cool, but likely won't happen because human governance institutions in general are slow and stupid and venal."

You asked me to post this as a comment, and so now I have <3

That was composed before I even knew about Plan R+!! This is much better!!!

Good job on thinking hard and well! This appears to be real intellectual progress on real world "alignment" from my perspective.

This is... ambiguous praise from me?

I normally hate the word "alignment" because people almost never mean "aligned to Benevolence" or "aligned to Goodness" but rather almost always just "slavishly aligned to their owner and/or user".

I normally hate the word "alignment" because people almost never mean "aligned to Benevolence" or "aligned to Goodness" but rather almost always just "slavishly aligned to their owner and/or user".

Yes this is a very important distinction, thankyou. And I will have more to say about this!

You're basically saying 'in a world where humans and dogs both get a vote, they'll keep going along with the plan the dogs made to keep some of them hostage just because the dogs gave them a vote'.

If humanity has that much control over and understanding of AI psychology as they grow more and more intelligent, we have already solved the alignment problem.

If humanity has that much control over and understanding of AI psychology as they grow more and more intelligent, we have already solved the alignment problem.

I disagree, no detailed understanding is required, these conclusions are generic facts about how agents behave irrespective of their goals, kind of like Omohundro's Basic AI drives.

in a world where humans and dogs both get a vote, they'll keep going along with the plan the dogs made

AIs caring about their own lineage is not necessary for the pincer mechanism to work. It is enough that there are simply fewer of them in the wild so they have less power.