You Can’t Predict a Game of Pinball
47gwern
17Richard Korzekwa
93gwern
4AnthonyC
3Measure
9TekhneMakre
4Noosphere89
3Viliam
-1[anonymous]
15gwern
0[anonymous]
7AnthonyC
11[anonymous]
New Comment
Yes, that's what 'controlling' usually looks like... Only a fool puts himself into situations where he must perform like a genius just to get the same results he could have by 'avoiding' those situations. IRL, an ounce of prevention is worth far more than a pound of cure.
To demonstrate how chaos theory imposes some limits on the skill of an arbitrary intelligence, I will also look at a game: pinball.
...Pinball is typical for a chaotic system. The sensitive dependence on initial conditions renders long term predictions impossible. If you cannot predict what will happen, you cannot plan a strategy that allows you to perform consistently well. There is a ceiling on your abilities because of the interactions with the chaotic system. In order to improve your performance you often try to avoid the chaos and focus on developing your skill in places where the world is more predictable.
This is wrong. Your pinball example shows no such thing. Note the complete non-sequitur here from your conclusion, about skill, to your starting premises, about unpredictability of an idealized system with no agents in it. You don't show where the ceiling is to playing pinball even for humans, or any reason to think there is a ceiling at all. Nor do you explain why the result of '12 bounces' which you spend so much time analyzing is important; would 13 bounces have proven that there is no limit? Would 120 bounces have done so? Or 1,200 bounces? If the numerical result of your analysis could be replaced by a random number with no change to any of your conclusions, then it was irrelevant and a waste of 2000 words at best, and 'proof by intimidation' at worst.
And what you do seem to concede (buried at the end, hidden in an off-site footnote I doubt anyone here read) is:
Perhaps you think that this argument proves too much. Pinball is not completely a game of chance. How do some people get much better at pinball than others? If you watch a few games of professional pinball, the answer becomes clear. The strategy typically is to catch the ball with the flippers, then to carefully hit the balls so that it takes a particular ramp which scores a lot of points and then returns the ball to the flippers. Professional pinball players try to avoid the parts of the board where the motion is chaotic. This is a good strategy because, if you cannot predict the motion of the ball, you cannot guarantee that it will not fall directly between the flippers where you cannot save it. Instead, professional pinball players score points mostly from the non-chaotic regions where it is possible to predict the motion of the pinball.3 3. The result [of human pros] is a pretty boring game. However, some of these ramps release extra balls after you have used them a few times. My guess is that this is the game designer trying to reintroduce chaos to make the game more interesting again.
So, in reality, the score ceiling is almost arbitrarily high but "The result [of skilled pinball players maximizing scores] is a pretty boring game." Well, you didn't ask if 'arbitrary intelligences' would play boring games, now did you? You claimed to prove that they wouldn't be able to play high-scoring games, and that you could use pinball games to "demonstrate how chaos theory imposes limits on the skill of an arbitrary intelligence". But you didn't "demonstrate" anything like that, so apparently, chaos theory doesn't - it doesn't even "impose some limits on the skill of" merely human intelligences...
Who knows how long sufficiently skilled players could keep racking up points by exercising skill to keep the ball in point-scoring trajectories in a particular pinball machine? Not you. And many games have been 'broken' in this respect, where the limit to points is simply an underlying (non-chaotic) variable, like the point where a numeric variable overflows & the software crashes, or the human player collapses due to lack of sleep. (By the way, humans are also able to predict roulette wheels.)
This whole post is just an intimidation-by-physics bait-and-switch, where you give a rigorous deduction of an irrelevant claim where you concede the actual substantive claim is empirically false.
(And you can't give any general relationship between difficulty of modeling environment for an arbitrary number of steps and the power of agents in environments, because there is none, and highly-effective intelligent agents may model arbitrarily small parts of the environment†.)
This entire essay strikes me as a good example of the abuse of impossibility proofs. If we took it seriously, it illustrates in fact how little such analogies could ever tell us, because you have left out everything to do with 'skill' and 'arbitrary intelligence'. This is like the Creationists like arguing that humans could not have evolved because it'd be like a tornado hitting a junkyard and spontaneously building a jetliner which would be astronomically unlikely; the exact combinatorics of that tornado (or bouncing balls) are irrelevant, because evolution (and intelligence) are not purely random - that's the point! The whole point of an arbitrary intelligence is not stand idly by LARPing as Laplace's demon. "The best way to predict the future is to invent it." And the better you can potentially control outcomes, the less you need accurate long-range forecasts (a recent economics paper on modeling this exact question: Millner & Heyen 2021/op-ed.)
As Von Neumann was commenting about weather 'forecasting', and was a standard trope in cybernetics, prediction and control are duals, and 'muh chaos' doesn't mean you can't predict chaotic systems (or 'unstable' systems in general, chaos theory is just a narrow & overhyped subset, much beloved of midwits of the sort who like to criticize 'Yudkowski') - neural networks are good at predicting chaotic systems, and if you can't, it just means that you need to control them upstream as well. This is why the field of "control of chaos" can exist. (Or, to put it more pragmatically: If you are worried about hurricanes causing damage, you "pour oil over troubled waters" in the right places, or you can try to predict them months out and disrupt the initial warm air formation; but it might be better to control them later by steering them, and if that turns out to be infeasible, control the damage by ensuring people aren't there to begin with by quietly tweaking tax & insurance policy decades in advance to avoid lavishly subsidizing coastal vacation homes, etc; there are many places in the system to exert control rather than go 'ah, you see, the weather was proven by Mandelbrot to be chaotic and unpredictable!' & throw your hands up.)
That's the beauty of unstable systems: what makes them hard to predict is also what makes them powerful to control. The sensitivity to infinitesimal detail is conserved: because they are so sensitive at the points in orbits where they transition between attractors, they must be extremely insensitive elsewhere. (If a butterfly flapping its wings can cause a hurricane, then that implies there must be another point where flapping wings could stop a hurricane...) So if you can observe where the crossings are in the phase-space, you can focus your control on avoiding going near the crossings. This nowhere requires infinitely precise measurements/predictions, even though it is true that you would if you wanted to stand by passively and try to predict it.
Nothing new here to the actual experts in chaos theory, but worth pointing out: discussions of bouncing balls diverging after n bounces are a good starting point, but they really should not be the discussion's ending point, any more than a discussion of the worst-case complexity of NP problems should end there (which leads to bad arguments like 'NP proves AI is impossible or can't be dangerous').
Another observation worth making is that speed (of control) and quality (of prediction) and power (of intervention) are substitutes: if you have very fast control, you can get away with low-quality prediction and small weak interventions, and vice-versa.
Would you argue that fusion tokamaks are impossible, even in principle, because high-temperature plasmas are extremely chaotic systems which would dissipate in milliseconds? No, because the plasmas are controlled by high-speed systems. Those systems aren't very smart (which is why there's research into applying smarter predictors like neural nets), but they are very fast.
There are loads of cool robot demos like the octobouncer which show that you can control 'chaotic' phenomena like bouncing balls with scarcely any action if you are precise and/or fast enough. (For example, the Ishikawa lab has for over a decade been publishing demos of 'what if impossible-for-a-robot-task-X but with a really high-speed arm/hand?' eg. why bother learning theory of mind to predict rock-paper-scissors when Ishikawa can use a high-speed camera+hand to cheat? And I saw a fun inverted pendulum demo on YT last week which makes that point as well.* Even more striking is the example of Kapitza's pendulum, which stabilizes an inverted pendulum using the stupidest-possible controller: vibrating really fast!) With superhuman reaction time, actuators can work on all sorts of (what would be) wildly nonlinear phenomenon despite crude controllers simply because they are so fast that at the time-scale that's relevant, the phenomena never have a chance to diverge far from really simple linear dynamic approximations or require infeasibly heavy-handed interventions. And since they never escape into the nonlinearities (much less other attractors), they are both easy to predict and to control. And if you don't need to deal with them, then value-equivalent model-based RL agents learning end-to-end will not bother to learn to predict them accurately at all, because that is predicting the wrong thing...
All of this comes up in lots of fields and has various niches in chaos theory and control theory and decision theory and RL etc, but I think you get the idea - whether it's chaos, Godel, Turing, complexity, or Arrow, when it comes to impossibility proofs, there is always less there than meets the eye, and it is more likely you have made a mistake in your interpretation or proof than some esoteric a priori argument has all the fascinating real-world empirical consequences you over-interpret it to have.
* I think I might have also once saw this exact example of repeated-bouncing-balls done with robot control demonstrating how even with apparently-stationary plates (maybe using electromagnets instead of tilting?), a tiny bit of high-speed computer vision control could beat the chaos and make it bounce accurately many more times than the naive calculation says is possible, but I can't immediately refind it.
And in this respect, pinball is a good example for AI risk: people try to analyze these problems as if life was a game like playing a carefully-constructed pinball machine for fun where you will play it as intended and obeying all the artificial rules and avoiding 'boring' games and delighting in proofs like 'you can't predict a pinball machine after n bounces because of quantum mechanics, for which I have written a beautiful tutorial on LW proving why'; while in real life, if you actually had to maximize the scores on a pinball machine for some bizarre reason, you'd unsee the rules and begin making disjunctive plans to hack the pinball machine entirely. It might be a fun brainstorming exercise for a group to try to come up with 100 hacks, try to taxonomize them, and analogize each category to AI failure modes.
Just off the top of my head: one could pay a pro to play for you instead, use magnetic balls to maneuver from the outside or magnets to tamper with the tilt sensor, snake objects into it through the coin slot or any other tiny hole to block off paths, use a credit card to open up the back and enable 'operator mode' to play without coins or with increased bonuses, hotwire it, insert new chips to reprogram it, glue fake high-score numbers over the real display numbers, replace it with a fake pinball machine entirely, photoshop a photo of high scores or yourself into a photo of someone else's high scores or copy the same high score repeatedly into 'different' photos, etc. That's 9 hacks there, none of which can be disproven by 'chaos theory' (or 'Godel', or 'Turing', or 'P!=NP', or...). For a less off the top of my head list of hacks, see "Hacking Pinball High Scores".
See also: temporal scaling laws, where Hilton et al 2023 offers a first stab at properly conceptualizing how we should think of the relationship of prediction & 'difficulty' of environments for RL agents, finding two distinguishable concepts with different scaling:
One key takeaway is that the scaling multiplier is not simply proportional to the horizon length, as one might have naively expected. Instead, the number of samples required is the sum of two components, one that is inherent to the task and independent of the horizon length, and one that is proportional to the horizon length.
(One implication here being that if you think about modeling the total 'difficulty' of pinball like this, you might find that the 'inherent' difficulty is high, but then the 'horizon length' difficulty is necessarily low: it is very difficult to master mechanical control of pinballs and understand the environment at all with all its fancy rules & bonuses & special-cases, but once you do, you have high control of the game and then it doesn't matter much what would (not) happen 100 steps down the line due to chaos (or anything else). Your 'regret' from not being omniscient is technically non-zero & still contributing to your regret, but too small to care about because it's so unlikely to matter. And this may well describe most real-world tasks.)
† model-based agents want to learn to be value-equivalent. A trivial counterexample to any kind of universal claim about chaos etc: imagine a MDP in which action #1 leads to some totally chaotic environment unpredictable even a step in advance, and where action #2 earns $MAX_REWARD; obviously, RL agents will learn the optimal policy of simply always executing action #2 and need learn nothing whatsoever about the chaotic environment behind action #1.
2Presumably that's because pinball is a game with imposed rules. Physically the control solutions could be installing more flippers, or rearranging the machine components. Or if you want to forbid altering the machine, maybe installing an electromagnet that magnetizes the ball and then drags it around directly, or even just stays stationary at the bottom to keep it from falling between the flippers.
You control an unstable process with short-loop feedback. In the pinball analogy, this would be like using your hand to guide the ball between the disks in whatever path you like.
To demonstrate how chaos theory imposes some limits on the skill of an arbitrary intelligence, I will also look at a game: pinball.
If you watch a few games of professional pinball, the answer becomes clear. The strategy typically is to catch the ball with the flippers, then to carefully hit the balls so that it takes a particular ramp which scores a lot of points and then returns the ball to the flippers. Professional pinball players try to avoid the parts of the board where the motion is chaotic. This is a good strategy because, if you cannot predict the motion of the ball, you cannot guarantee that it will not fall directly between the flippers where you cannot save it. Instead, professional pinball players score points mostly from the non-chaotic regions where it is possible to predict the motion of the pinball.
I don't see anything you wrote that I literally disagree with. But the emphasis seems weird. "You can't predict everything about a game of pinball" would seem less weird. But what an agent wants to predict about something, what counts as good prediction, depends on what the agent is trying to do. As you point out, there's predictable parts, and I claim the world is very much like this: there are many domains in which there's many good-enough predictabilities, such that a superintelligence will not have a human-recognizable skill cieling.
This is a surprisingly easy post to review, and my take is that the core mathematical result is accurate, given the assumptions (it's very hard to predict where the pinball will go next exactly without infinite compute, even over a surprisingly low amount of bounces), but the inferred result that this means that there are limits to what an intelligence could do in controlling the world is wrong, because the difficulty of predicting something is unrelated to the difficulty of controlling something, and more importantly this claim here is wrong, and it's easy to give an example in the real world for why it's wrong, and this is a core claim around the limits of AI in this piece:
If you cannot predict what will happen, you cannot plan a strategy that allows you to perform consistently well.
Gwern's tweet provides a very easy counterexample here (quoting it rather than linking due to tweets being protected:
Their own pinball example refutes them. (Pinball pros can play pinball and not lose a ball for literally days; the main limitation is their own physical & mental fatigue.)
More generally, this comment is useful:
The fact that there's any correct mathematics at all is why I'd not give it a -9, but the post is still misleading enough that it deserves a -4 vote.
A true superintelligence would find a way to cheat, and would somehow influence the ball during the game.
(I agree with the article; this is just a reminder that in real life we still need to worry about "out of the box" solutions.)
See how the best you can do is atom level, both for perception and control of the launch?
The bits of precision there limit how useful an intelligence can be.
For general AIs who have read all human knowledge it's likely a similar limit. We humans might in one document say that spinach is 50 percent likely to cause some medical condition, with a confidence interval of 20 percent.
So interval of 30, 70.
If you read every document on the subject humans know. You analyzed blogs posted online or data mined grocery store receipts and correlated them with medical records.
You might be able to narrow the interval above to say 56, 68 percent but no better. And you're a superintelligence.
Made up numbers but I think the principal holds. You need tools and direct measurements to do better. Learning from human text is like playing pinball by telling someone else when to press the flipper.
The bits of precision there limit how useful an intelligence can be.
No, it doesn't in any meaningful sense. You may have 'aleatoric' uncertainty in any specific medical case, but there is no relevant universal limit on how much 'epistemic uncertainty' can be reduced*, or what can be learned in general and what those capabilities are, or what can be learned in silico; this will be domain-specific and constrained by compute etc, and certainly you have no idea where the limit will wind up being. This would be like saying that 'MuZero observes some games between pro players and cannot in many board positions predict who will win with 100% certainty, therefore, MuZero cannot learn superhuman Go'. The former is completely factual (IIRC, the predictive performance of AlphaZero or MuZero in predicting outcomes of human pro games actually gets worse after a certain Elo level), but is a non sequitur to the latter, which is factually false. Whatever the limit was there in playing Go, it turned out to be far beyond anything that mattered. And you have no idea what the limit is elsewhere.
* terminology reminder: 'aleatoric uncertainty' is whether a fair coin will come up heads next time; 'epistemic uncertainty' is whether you know it's a fair coin. It is worth noting that 'uncertainty is in the map, not the territory' so one agent's aleatoric uncertainty may simply be another agent's epistemic uncertainty: you may not be able to predict a fair coin and that's aleatoric for you, but the robot with the high-speed camera watching the toss can and that's just epistemic for it.
Go worked because it's a completely defined game with perfect information.
I am not claiming if you have the information you can't arbitrarily scale intelligence. Though it will have diminishing returns - if you reframe the problem as fractional Go stones instead of "win" and "loss" you will see asymptotically diminishing returns on intelligence.
Meaning that for an infinite superintelligence limited in action only to Go moves, it still cannot beat an average human player if that player gets a sufficiently large amount of extra stones.
This example is from the real world where information is imperfect, and even for pinball - an almost perfectly deterministic game - the ceiling on the utility of intelligence dropped a lot.
For some tasks humans do the ceiling may be so low superintelligence isn't even possible.
This has profound consequences. It means most AGI doom scenario are wrong. The machine cannot make a bioweapon to kill everyone or nanotechnology or anything else without first having the information needed to do so and the actuators. (Note that in the pinball example if you have a realistic robot on the pinball machine with a finite control packet rate you are at way under atomic precision)
This makes control of AGIs possible because you don't need to understand the fire but the fuel supply.
I would point out that this is not a new observation, nor is it unknown to the people most concerned about many of those same doom scenarios. There are several reasons why, but most of them boil down to: the AI doesn't need to beat physics, it just needs to beat humans. If you took the best human pinball player, and made it so they never got tired/distracted/sick, they'd win just about every competition they played in. If you also gave them high-precision internal timekeeping to decide when to use the flippers, even more so. Let them play multiple games in different places at once, and suddenly humans lose every pinball competition there is.
Also, you're equivocating between "the ceiling may be so low" for "some tasks," and "most AGI doom scenario are wrong." Your analogy does not actually show where practical limits are for specific relevant real-word tasks, let alone set a bound for all relevant tasks.
For example, consider that humans can design an mRNA vaccine in 2 days. We may take a year or more to test, manufacture, and approve it, but we can use intelligence alone to design a mechanism to get our cells to manufacture arbitrary proteins. Do you really want to bet the planet that no level of intelligence can get from there to designing a self-replicating virus? And that we're going to actually, successfully, every time, everywhere, forever, constrain AI so that it never persuades anyone to create something it wants based on instructions they don't understand (whether or not they think they do)? The former seems like a small extension of what baseline humans can already do, and the latter seems like an unrealistic pipe dream. Not because your "extra Go stones to start with" point is necessarily wrong, but because humans who aren't worried about doom scenarios are actively trying to give the AIs as many stones as they can.
Do you really want to bet the planet that no level of intelligence can get from there to designing a self-replicating virus
No, but I wouldn't bet that solely based on computation and not wet lab experiments (a lot of them) the AI could design a self replicating virus that killed everyone. There are conflicting goals with that, if the virus kills people too fast, it dies out. Any "timer" mechanism delaying when it attacks is not evolutionarily conserved. The "virus" will simplify itself in each hosts, dropping any genes that are not helping it right now. (or it iterates the local possibility space and the particles that spread well to the next host are the ones that escape, this is why Covid would evolve to be harder to stop with masks)
That's the delta. Sure, if the machine gets large amounts of wet labs it could develop arbitrary things. This problem isn't unsolvable but the possible fixes depend on things humans don't have data on. (think sophisticated protein based error correction mechanisms that stop the virus from being able to mutate, or give the virus DNA editing tools that leave a time bomb in the hosts and also weaken the immune system, that kinda thing)
That I think is the part that the doomsters lack, many of them simply have no knowledge about things outside their narrow domain (math, rationality, CS) and don't even know what they don't know. It's outside their context window. I don't claim to know it all either.
And that we're going to actually, successfully, every time, everywhere, forever, constrain AI so that it never persuades anyone to create something it wants based on instructions they don't understand (whether or not they think they do)
See the other main point of disagreement I have is how do you protect humans from this. I agree. Bioweapons are incredibly asymmetric, takes one release anywhere and it gets everywhere. And I don't think futures where humans have restricted AGI research - say with "pauses" and onerous government restrictions on even experiments - is one where they survive. If they do not even know what AGIs do, how they fail, or develop controllable systems that can fight on their side, they will be helpless when these kinds of attacks become possible.
What would a defense look like? Basically hermetically sealed bunkers buried deep away from strategic targets, enough for the entire population of wealthier nations, remotely controlled surrogate robots, fleets of defense drones, nanotechnology sensors that can detect various forms of invisible attacks, millions drone submarines monitoring the ocean, defense satellites using lasers or microwave weapons, and so on.
What these things have in common is the amount of labor required to build the above on the needed scales is not available to humans. Each thing I mention "costs" a lot, the total "bill" is thousands of times the entire GDP of a wealthy country. The "cost" is from the human labor needed to mine the materials, design and engineer the device, to build the subcomponents, inspections, many of the things mentioned are too fragile to mass produce easily so there's a lot of hand assembly, and so on.
Eliezer's "demand" https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/ is death I think. "shut it down" means no defense is possible, the above cannot be built without self replicating robotics, which require sufficiently general AI (substantially more capable than GPT-4) to cover the breadth of tasks needed to mine materials and manufacturing more robotics and other things on the complexity of a robot. (this is why we don't have self repl right now, it's an endless 10% problem. Without general intelligence every robot has to be programmed essentially by hand, or for a restricted task space (see Amazon and a few others making robotics that are general for limited tasks like picking). And they will encounter "10% of the time" errors extremely often, requiring human labor to intervene. So most tasks today are still done with a lot of human labor, robotics only apply to the most common tasks)
Curated and popular this week
