While chaos does place limits on what an ASI could do, those limits don’t prevent it from outsmarting humanity and taking over.  

First, a recap. Chaos theory says that there are some systems which become totally unpredictable over finite timescales. Not necessarily as a result of true randomness, but as a result of small differences in initial conditions being amplified. Some examples of chaotic systems include: 

1. the weather, which can’t be predicted more than two weeks into the future,
2. hard sudoku puzzles which can be shown equivalent to some chaotic systems,
3. and pinball, where “the uncertainty in the location of the pinball grows by a factor of about 5 every time the ball collides with one of the disks”. 

People have argued that this places bounds on a superintelligence’s predictive abilities. So they can’t be much more clever than humans and do things like outfight all of humanity. For instance, since pinball is chaotic, that implies there are limits to even a superintelligence’s ability to play the game. 

A common issue with this class of arguments is that they tend to prove too much. Their logical structure could be replicated exactly, with all mention of ASI replaced with humans, for instance. For instance, take the following simplified argument. Chaos is common. In order to come up with plans to achieve goals, one needs to predict it well. Chaotic systems are unpredictable. Moreover, they are common. Therefore, one can only plan to a limited extent. Therefore, ASI is impossible. This argument works perfectly well if one replaces “ASI” at the end with “humans”. 

Proponents of these arguments do not believe humans are impossible. Rather, humans have mined all the great veins of regularity in the world. 

Let us grant that it is the case that the above arguments do not rule out the existence of humanity. It is still the case that arguments based on chaos theory are lacking. 

The key question here is whether humanity has already mined the world for all its predictable structure and what is left is basically pure, unpredictable, uncontrollable^[1] chaos. If this were true, then when an ASI attempts a takeover, it would inevitably run into the task of predicting some chaotic system. 

This is a very difficult claim to show. For one reason, any argument for this would need to be weak enough that it didn’t rule out continued human progress, which we anticipate to be quite large. For another, one would need to rule out every possible pathway to route around, or alter, chaotic systems. 

Absent arguments of this form, one is left with some prior over “how much regularity there is left to mine.” There seems to be no principled reason to assume the prior should be narrowly concentrated about “not much”. 

1. ^{^}

    “What we cannot predict, we shall control” - John Von Neuman. Chaotic systems, like some instances of the double pendulum, can be made predictable through the influence of external control mechanisms. For instance, a high frequency vibration is enough to force a double pendulum into a predictable, upright state.