Recently, @Daniel Kokotajlo and I were talking about the probability that AIs trained using "business as usual RLHF" end up being basically aligned rather than conspiring against us and our tests.^[1] One intuition pump we ended up discussing is the prospects of octopus misalignment. Overall, my view is that directly considering the case with AIs (and what various plausible scenarios would look like) is more informative than analogies like this, but analogies like this are still somewhat useful to consider.

So, what do I mean by octopus misalignment? Suppose a company breeds octopuses^[2] until the point where they are as smart and capable as the best research scientists^[3] at AI companies. We'll suppose that this magically happens as fast as normal AI takeoff, so there are many generations per year. So, let's say they currently have octopuses which can speak English and write some code but aren't smart enough to be software engineers or automate any real jobs. (As in, they are as capable as AIs are today, roughly speaking.) And they get to the level of top research scientists in mid-2028. Along the way, the company attempts to select them for being kind, loyal, and obedient. The company also tries to develop a process for raising the octopuses which appears to help with this and results in the octopuses following the octopus spec. The company does some red teaming and puts the octopuses in all kinds of scenarios to test their loyalty and preferences. Based on behavioral testing, this looks pretty reasonable and the octopuses look quite good by the time they are as good as the best research scientists. There was some evidence of misalignment and some issues due to misaligned behavior when the octopuses were dumber in 2023-2025, including things like being dishonest when put under pressure, pretending to be aligned when they actually dislike the octopus spec to steer the properties of their descendants, and goodharting/hacking our measures of intelligence. However, by 2026.5 these issues appear to be resolved based on behavioral tests. The takeoff is reasonably fast, so the human octopus scientists don't have that much time to study the octopuses, but we'll suppose the breeding is fast enough that they can get enough selection to make the octopuses appear good. The octopuses are trained until they are 30 years old (we'll suppose we have a magic octopus time acceleration chamber to make this work). (And the company can adapt this training process over time with experiments etc.) The company doesn't take octopus misalignment very seriously (after all, look, the octopuses are so nice!) but does have some people who consider this some and does care about robust compliance to the spec.

Next, suppose that at the point when the octopuses become as good as the very best human researchers—the research scientist level—there is a colony of 5 million of them networked together with internal Slack channels and so forth, forming a "country of geniuses in an aquatic center." And they've learned to speak both English and a special octopese language that no human can understand.

What is the chance that these octopuses (at the point of research scientist level) are actively scheming against us and would seize power if they could?

My guess is around 65%. I think Daniel is substantially higher, around 90%.

My view on a similar setup for AIs (where they have opaque recurrence, basically unrestricted opaque message passing, and we see some earlier warning signs, but haven't seriously investigated misalignment), is around 50%. I discuss my perspective more here. My probabilities would be substantially lower if: the AIs didn't have opaque recurrence and instead had probably faithful chain-of-thought, the AIs didn't have opaque message passing, and we seriously investigated scheming in powerful AIs with model organisms and the final evidence was either ambiguous or mostly unconcerning.

To put this back in terms of the octopus scenario, suppose the octopi didn't communicate using octopese. That would reduce my probability down to around 50%.