Optimistic Assumptions, Longterm Planning, and "Cope"
38Eli Tyre
6Seth Herd
3Eli Tyre
2Raemon
18[anonymous]
6Noosphere89
5Seth Herd
2hairyfigment
10Richard_Ngo
23Eli Tyre
16Raemon
15localdeity
4Raemon
4aysja
0Richard_Ngo
3Nathan Helm-Burger
2Seth Herd
1Amalthea
3Richard_Ngo
5Raemon
1Amalthea
7dirk
11Dweomite
3Nathan Helm-Burger
6lemonhope
4CronoDAS
6Jay
4Raemon
5Jay
1[comment deleted]
4trevor
4Stefan De Young
9Raemon
4Noosphere89
3Dweomite
6Noosphere89
6Dweomite
4Seth Herd
2CronoDAS
2Noosphere89
2Seth Herd
4Dweomite
3Raemon
2Tapatakt
1Dweomite
3pom
1Review Bot
New Comment
It seems important to note that the posited reasoning flaw is much more rational in the context of a puzzle game. In that context, you know that there's a correct answer. If you've exhaustively searched through the possibilities, and you're very confident that there's no solution unless assumption X holds...that's actually a pretty good reason to believe X.
A syllogism:
P1: There is a solution.
P2: If not X, there would be no solution.
C: Therefore X.
In this context, subjects seem to be repeatedly miscalibrated about P2. It turns out that they had searched through the space of possibilities much less exhaustively than they thought they had. So they were definitely wrong, overall.
But this syllogism is still valid. And P1 is a quite reasonable assumption.
There's an implication that people are applying an identical pattern of reasoning to plans for AI risk. In that context, not only is P2 non-obvious for any given version of P2, but P1 isn't solid either: there's no guarantee that there's a remotely feasible solution in scope, given the available resources.
In that case, if indeed people are reasoning this way, they're making two mistakes, both PI and P2, whereas in puzzle game context, people are only really making one.
From my observations of people thinking about AI risk, I also think that people are (emotionally) assuming that there's a viable solution AI risk, and their thinking warps accordingly.
But it seems important to note that if we're drawing the conclusion that people make that particular error, from this dataset, you're attributing a reasoning error to them in one context, based on observing them use a similar assumption in a different context in which that assumption is, in fact, correct.
2Yes they're correct in assuming success is possible in those situations - but their assumption of the possible routes to success is highly incorrect. People are making a very large error in overestimating how well they understand the situation and failing to think about other possibities. This logical error sounds highly relevant to alignment and AI risk.
Eliezer Yudkowsky periodically complains about people coming up with questionable plans with questionable assumptions to deal with AI, and then either:
- Saying "well, if this assumption doesn't hold, we're doomed, so we might as well assume it's true."
- Worse: coming up with cope-y reasons to assume that the assumption isn't even questionable at all. It's just a pretty reasonable worldview.
Sometimes the questionable plan is "an alignment scheme, which Eliezer thinks avoids the hard part of the problem." Sometimes it's a sketchy reckless plan that's probably going to blow up and make things worse.
Some people complain about Eliezer being a doomy Negative Nancy who's overly pessimistic.
This is somewhat beyond the scope of the primary point I think you are trying to make in this post, but I suspect this recurring dynamic (which I have observed myself a number of times) is more caused by (and logically downstream of) an important object-level disagreement about the correctness of the following claim John Wentworth makes in one of the comments on his post on how "Most People Start With The Same Few Bad Ideas":
I think a lot of people don't realize on a gut level that a solution which isn't robust is guaranteed to fail in practice. There are always unknown unknowns in a new domain; the presence of unknown unknowns may be the single highest-confidence claim we can make about AGI at this point. A strategy which fails the moment any surprise comes along is going to fail; robustness is necessary. Now, robustness is not the same as "guaranteed to work", but the two are easy to confuse. A lot of arguments of the form "ah but your strategy fails in case X" look like they're saying "the strategy is not guaranteed to work", but the actually-important content is "the strategy is not robust to <broad class of failures>"; the key is to think about how broadly the example-failure generalizes. (I think a common mistake newcomers make is to argue "but that particular failure isn't very likely", without thinking about how the failure mode generalizes or what other lack-of-robustness it implies.)
(Bolding is mine.) It seems to me that, at least in a substantial number of cases in which such conversations between Eliezer and some other alignment researcher who presents a possible solution to the alignment problem, Eliezer is trying to get this idea across and the other person genuinely disagrees[1]. To give a specific example of this dynamic that seems illustrative, take Quintin Pope's comment on the same post:
The issue is that it's very difficult to reason correctly in the absence of an "Official Experiment"[1]. I think the alignment community is too quick to dismiss potentially useful ideas, and that the reasons for those dismissals are often wrong. E.g., I still don't think anyone's given a clear, mechanistic reason for why rewarding an RL agent for making you smile is bound to fail (as opposed to being a terrible idea that probably fails).
[1] More precisely, it's very difficult to reason correctly even with many "Official Experiments", and nearly impossible to do so without any such experiments.
- ^
As opposed to simply being unaware of this issue and becoming surprised when Eliezer brings it up.
Plans obviously need some robustness to things going wrong, and in a sense I agree with John Wentworth, if weakly, that some robustness is a necessary feature of a plan, and some verification is actually necessary.
But I have to agree that there is a real failure mode identified by moridinamael and Quintin Pope, and that is perfectionism, meaning that you discard ideas too quickly as not useful, and this constraint is the essence of perfectionism:
I have an exercise where I give people the instruction to play a puzzle game ("Baba is You"), but where you normally have the ability to move around and interact with the world to experiment and learn things, instead, you need to make a complete plan for solving the level, and you aim to get it right on your first try.
It asks for both a complete plan to solve the whole level, and also asks for the plan to work on the first try, which outside of this context implies either the problem is likely unsolvable or you are being too perfectionist with your demands.
In particular, I think that Quintin Pope's comment here is genuinely something that applies in lots of science and problem solving, and that it's actually quite difficult to reasoin well about the world in general without many experiments.
2I think the concern is that, if plans need some verification, it may be impossible to align smarter-than-human AGI. In order to verify those plans, we'd have to build one. If the plan doesn't work (isn't verified), that may be the end of us - no retries possible.
There are complex arguments on both sides, so I'm not arguing this is strictly true. I just wanted to clarify that that's the concern and the point of asking people to solve it on the first try. I think ultimately this is partly but not 100% true of ASI alignment, and clarifying exactly how and to what degree we can verify plans empirically is critical to the project. Having a plan verified on weak, non-autonomous/self-aware/agentic systems may or may not generalize to that plan working on smarter systems with those properties. Some of the ways verification will or won't generalize can probably be identified with careful analysis of how such systems will be functionally different.
1https://arxiv.org/abs/1712.05812
It's directly about inverse reinforcement learning, but that should be strictly stronger than RLHF. Seems incumbent on those who disagree to explain why throwing away information here would be enough of a normative assumption (contrary to every story about wishes.)
These are interesting anecdotes but it feels like they could just as easily be used to argue for the opposite conclusion.
That is, your frame here is something like "planning is hard therefore you should distrust alignment plans".
But you could just as easily frame this as "abstract reasoning about unfamiliar domains is hard therefore you should distrust doom arguments".
Also, the second section makes an argument in favor of backchaining. But that seems to contradict the first section, in which people tried to backchain and it went badly. The best way for them to make progress would have been to play around with a bunch of possibilities, which is closer to forward-chaining.
(And then you might say: well, in this context, they only got one shot at the problem. To which I'd say: okay, so the most important intervention is to try to have more shots on the problem. Which isn't clearly either forward-chaining or back-chaining, but probably closer to the former.)
1These are interesting anecdotes but it feels like they could just as easily be used to argue for the opposite conclusion.
That is, your frame here is something like "planning is hard therefore you should distrust alignment plans".
But you could just as easily frame this as "abstract reasoning about unfamiliar domains is hard therefore you should distrust doom arguments".
That doesn't sound right to me.
The reported observation is not just that these particular people people failed at a planning / reasoning task. The reported observation is that they repeatedly made optimistic miscalibrated assumptions, because those assumptions supported a plan.
There's a more specific reasoning error that's being posited, beyond "people are often wrong when trying to reason about abstract domains without feedback". Something like "people will anchor on ideas, if those ideas are necessary for the success of a plan, and they don't see an alternative plan."
If that posit is correct, that's not just an update of "reasoning abstractly is hard and we should widen our confidence intervals / be more uncertain". We should update to having a much higher evidential bar for the efficacy of plans.
3
3I had a second half of this essay that felt like it was taking too long to pull together and I wasn't quite sure who I was arguing with. I decided I'd probably try to make it a second post. I generally agree it's not that obvious what lessons to take.
The beginning of the second-half/next-post was something like:
There's an age-old debate about AI existential safety, which I might summarize as the viewpoints:
1. "We only get one critical try, and most alignment research dodges the hard part of the problem, with wildly optimistic assumptions."
vs
2. "It is basically impossible to make progress on remote, complex problems on your first try. So, we need to somehow factor the problem into something we can make empirical progress on."
I started out mostly thinking through lens #1. I've updated that, actually, both views are may be "hair on fire" levels of important. I have some frustrations with both some doomer-y people who seem resistant to incorporating lens #2, and with people who seem to (in practice) be satisfied with "well, iterative empiricism seems tractable, and we don't super need to incorporate frame #1)
I am interested in both:
- trying to build "engineering feedback loops" that more accurately represent the final problem as best we can, and then iterating on both "solving representative problems against our current best engineered benchmarks" while also "continuing to build better benchmarks. (Automating Auditing and Model Organisms of Misalignment seem like attempts at this)
- trying to develop training regimens that seem like they should help people plan better in Low-Feedback-Domains, which includes theoretic work, and empirical research that's trying to keep their eye on the longterm ball better, and the invention of benchmarks a la previous bullet.
That is, your frame here is something like "planning is hard therefore you should distrust alignment plans".
But you could just as easily frame this as "abstract reasoning about unfamiliar domains is hard therefore you should distrust doom arguments".
I think received wisdom in cryptography is "don't roll your own crypto system". I think this comes from a bunch of overconfident people doing this and then other people discovering major flaws in what they did, repeatedly.
The lesson is not "Reasoning about a crypto system you haven't built yet is hard, and therefore it's equally reasonable to say 'a new system will work well' and 'a new system will work badly'." Instead, it's "Your new system will probably work badly."
I think the underlying model is that there are lots of different ways for your new crypto system to be flawed, and you have to get all of them right, or else the optimizing intelligence of your rivals (ideally) or the malware industry (less ideally) will find the security hole and exploit it. If there are ten things you need to get right, and you have a 30% chance of screwing up each one, then the chance of complete success is 2.8%. Therefore, one could say that, if there's a general fog of "It's hard to reason about these things before building them, such that it's hard to say in advance that the chance of failure in each thing is below 30%", then that points asymmetrically towards overall failure.
I think Raemon's model (pretty certainly Eliezer's) is indeed that an alignment plan is, in large parts, like a security system in that there are lots of potential weaknesses any one of which could torpedo the whole system, and those weaknesses will be sought out by an optimizing intelligence. Perhaps your model is different?
Also, the second section makes an argument in favor of backchaining. But that seems to contradict the first section, in which people tried to backchain and it went badly.
This didn't come across in the post, but – I think people in the experiment were mostly doing things closer to (simulated) forward chaining, and then getting stuck, and then generating the questionable assumptions. (which is also what I tended to do when I first started this experiment).
An interesting thing I learned is that "look at the board and think without fiddling around" is actually a useful skill to have even when I'm doing the more openended "solve it however seems best." It's easier to notice now when I'm fiddling around pointlessly instead of actually doing useful cognitive work.
But you could just as easily frame this as "abstract reasoning about unfamiliar domains is hard therefore you should distrust doom arguments".
But doesn't this argument hold with the opposite conclusion, too? E.g. "abstract reasoning about unfamiliar domains is hard therefore you should distrust arguments about good-by-default worlds".
Yepp, all of these arguments can weigh in many different directions, depending on your background beliefs. That's my point.
No, failure in the face of strong active hostile optimization pressure is different from failure when acting to overcome challenges in a neutral universe that just obeys physical laws. Building prisons is a different challenge from building bridges. Especially in the case where you know that there will be agents both inside and outside the prison that very much want to get through. Security is inherently a harder problem. A puzzle which has been designed to be challenging to solve and to have a surprising unintuitive solution is thus a better match for designing a security system. I bet you'd have much higher success rate of 'first plans' if the game were one of those bridge building games.
Yes to your first point. I think that
abstract reasoning about unfamiliar domains is hard therefore you should distrust doom arguments
Is a fair characterization of those results. So would be the inverse, "abstract reasoning about unfamiliar domains is hard therefore you should distrust AI success arguments".
I think both are very true, and so we should distrust both. We simply don't know.
I think the conclusion taken,
planning is hard therefore you should distrust alignment plans
Is also valid and true.
People just aren't as smart as we'd like to think we are, particularly in reasoning about complex and unfamiliar domains. So both our plans and evaluations of them tend to be more untrustworthy than we'd like to think. Planning and reasoning require way more collective effort than we'd like to imagine. Careful studies of both individual reasoning in lab tasks and historical examples support this conclusion.
One major reason for this miscalibration is the motivated reasoning effect. We tend to believe what feels good (predicts local reward). Overestimating our reasoning abilities is one such belief among vary many examples of motivated reasoning.
Yes, I'm saying it's a reasonable conclusion to draw, and the fact that it isn't drawn here is indicative of a kind of confirmation bias.
So, I do think definitely I've got some confirmation bias here – I know because the first thing I thought when I saw was "man this sure looks like the thing Eliezer was complaining about" and it was awhile later, thinking it through, that was like "this does seem like it should make you really doomy about any agent-foundations-y plans, or other attempts to sidestep modern ML and cut towards 'getting the hard problem right on the first try.'"
I did (later) think about that a bunch and integrate it into the post.
I don't know whether I think it's reasonable to say "it's additionally confirmation-bias-indicative that the post doesn't talk about general doom arguments." As Eli says, the post is mostly observing a phenonenon that seems more about planmaking than general reasoning.
(fwiw my own p(doom) is more like 'I dunno man, somewhere between 10% and 90%, and I'd need to see a lot of things going concretely right before my emotional center of mass shifted below 50%')
Thanks for clarifying. I do agree with the broader point that one should have a sort of radical uncertainty about (e.g.) a post AGI world. I'm not sure I agree it's a big issue to leave that out of any given discussion though, since it shifts probability mass from any particular describable outcome to the big "anything can happen" area. (This might be what people mean by "Knightian uncertainty"?)
What I found most interesting was people literally saying the words out loud, multiple times "Well, if this [assumption] isn't true, then this is impossible" (often explicitly adding "I wouldn't [normally] think this was that likely... but..."). And, then making the mental leap all the way towards "70% that this assumption is true." Low enough for some plausible deniability, high enough to justify giving their plan a reasonable likelihood of success.
It was a much clearer instance of mentally slipping sideways into a more convenient world, than I'd have expected to get.
I think this tendency might've been exaggerated by the fact that they were working on a puzzle game; they know the levels are in fact solvable, so if a solution seems impossible it's almost certainly the case that they've misunderstood something.
Yes, your prior on the puzzle being actually unsolvable should be very low, but in almost all such situations, 70% seems way too high a probability to assign to your first guess at what you've misunderstood.
When I prove to myself that a puzzle is impossible (given my beliefs about the rules), that normally leads to a period of desperate scrambling where I try lots of random unlikely crap just-in-case, and it's rare (<10%) that anything I try during that desperate scramble actually works, let alone the first thing.
In the "final" level of Baba Is You, I was stuck for a long time with a precise detailed plan that solved everything in the level except that there was one step in the middle of the form "and then I magically get past this obstacle, somehow, even though it looks impossible." I spent hours trying to solve that one obstacle. When I eventually beat the level, of course, it was not by solving that obstacle--it was by switching to a radically different approach that solved several other key problems in entirely different ways. In hindsight, I feel like I should have abandoned that earlier plan much sooner than I did.
In mitigation: I feel that solutions in Baba Is You are significantly harder to intuit than in most puzzle games.
Yes, but they failed with much higher than 30% likelihood even still. So the tendency you are pointing out wasn't helping much. I do agree that it would be a better test to make multiple levels using the level editor, and have some of them literally be unsolvable. Then have someone have the choice of writing out a solution or writing out a proof for why there could be no solution. I think you'd find an even lower success rate in this harder version.
I love this post.
I think you forgot to mention an important prerequisite. It was reasonable to assume the prereq but still worth mentioning I think. You should be looking for the W, for the clear win. It's easy to just fart around and forget you were trying to make something happen. And in real life there are often bigger and clearer wins available than is immediately apparent. Often this takes much time and energy and creativity to see. Often the more important/urgent problem can be easier to solve. People tend to love tricky things and puzzles. It can be hard to learn to love easy victory. Pop the soccer-ball! Stab your opponent in the back while they sleep on Christmas night! Replace your sails with diesel motors! Solve your integral numerically! Use steel instead of wood! — This lesson of course well known here but it's hard to have a consuming conversation or intriguing post about it. I often forget this.
Sometimes, you might as well solve the Rubix Cube by peeling the stickers off and sticking them back on.
TvTropes calls this Cutting the Knot, after the story of Alexander and the Gordian Knot.
Almost everyone believes one and only one of the following statements:
- We have, or can soon develop, the technology to power society with clean energy at a modern standard of living, or
- Global warming is not a major problem (e.g. it won't damage agricultural productivity to famine-inducing levels).
Logically, these statements have nothing to do with each other. Either, neither, or both could be correct (I suspect neither). They are, in the words of the post, questionable assumptions.
See also Beyond the Peak – Ecosophia and Germany's economy struggles with an energy shock that's exposing longtime flaws | AP News.
"Slipping into a more convenient world" is a good way of putting it; just using the word "optimism" really doesn't account for how it's pretty slippy, nor how the direction is towards a more convenient world.
In your Baba-Is-You methodology, were test takers able to play through tutorial levels normally before they switched to plan-in-advance? Did you ask them if they had any familiarity with the game beforehand?
People varied in how much Baba-Is-You experience they had. Some of them were completely new, and did complete the first couple levels (which are pretty tutorial-like) using the same methodology I outline here, before getting to a level that was a notable challenge.
They actually did complete the first couple levels successfully, which I forgot when writing this post. This does weaken the rhetorical force, but also, the first couple levels are designed more to teach the mechanics and are significantly easier. I'll update the post to clarify this.
Some of them had played before, and were starting a new level from around where they left off.
What I take away from this is that they should have separated the utility from an assumption being true, from the probability/likelihood of an assumption being true, and indeed this shows some calibration problems.
There is slipping into more convenient worlds for reasons based on utility rather than evidence, which is a problem (assuming it's solvable for you.)
This is an important takeaway, but I don't think your other takeaways help as much as this one.
That said, this constraint IRL makes almost all real-life problems impossible for humans and AIs:
I have an exercise where I give people the instruction to play a puzzle game ("Baba is You"), but where you normally have the ability to move around and interact with the world to experiment and learn things, instead, you need to make a complete plan for solving the level, and you aim to get it right on your first try.
In particular, if such a constraint exists, then it's a big red flag that the problem you are solving is impossible to solve, given that constraint.
Almost all plans fail on the first try, even for really competent plans and humans, and outside of very constrained regimes, 0 plans work out on the first try.
Thus, if you are truly in a situation where you are encountering such constraints, you should give up on the problem ASAP, and rest a little to make sure that the constraint actually exists.
So while this is a fun experiment, with real takeaways, I'd warn people that constraining a plan to work on the first try and requiring completeness makes lots of problems impossible to solve for us humans and AIs.
One of Eliezer's essays in The Sequences is called Shut Up and Do the Impossible
I'm confident Eliezer would agree with you that if you can find a way to do something easier instead, you should absolutely do that. But he also argues that there is no guarantee that something easier exists; the universe isn't constrained to only placing fair demands on you.
My point isn't that the easier option always exists, or even that a problem can't be impossible.
My point is that if you are facing a problem that requires 1-shot complete plans, and there's no second try, you need to do something else.
There is a line where a problem becomes too difficult to productively work on, and that constraint is a great sign of an impossible problem (if it exists.)
AND your accurate assessment of the difficulty. The overconfidence displayed in this mini-experiment seems to result in part from people massively misestimating the difficulty of this relatively simple problem. That's why it's so concerning WRT alignment.
Not really, but they are definitely more few-shot than other areas, but thankfully getting 1 thing wrong isn't usually an immediate game-ender (though it is still to be avoided, and importantly this is why these 2 areas are harder than a lot of other fields).
Ah- well said. I understand the rest of your comments better now. And I thoroughly agree, with a caveat about the complexity of the problem and the amount of thought and teamwork applied (e.g., I expect that a large team working for a month in effective collaboration would've solved the problem in this experiment, but alignment is probably much more difficult than that).
I'll get 10 extra units of production, or damage. But, then I reach the next stage, and it turns out I really needed 100 extra units to survive.
I'm having a hard time thinking of any of my personal experiences that match this pattern, and would be interested to hear a couple examples.
(Though I can think of several experiences along the lines of "I got 10 points of armor, and then it turned out the next stage had a bunch of attacks that ignore armor, so armor fundamentally stops working as a strategy." There are a lot of games where you just don't have the necessary information on your first try.)
Games I was particularly thinking of were They Are Billions, Slay The Spire. I guess also Factorio although the shape of that is a bit different.
(to be clear, these are fictional examples that don't necessarily generalize, but, when I look at the AI situation I think it-in-particular has an 'exponential difficulty' shape)
I haven't played They Are Billions. I didn't have that experience in Slay the Spire, but I'd played similar games before. I suppose my first roguelike deckbuilder did have some important combo-y stuff that I didn't figure out right away, although that game was basically unwinnable on your first try for a bunch of different reasons.
I am slightly intrigued by this game, as it seems to look like how I approach things in general when I have enough time and information to do so. Just an interesting aside, I like that people are also doing these kinds of things after all, or at least attempting to. As I am by no means "successful" with my method, it does seem to be the only way I can get myself to undertake something when it is important to me that I could at least have a somewhat realistic chance at succeeding.
The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.
Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?