Thanks for the careful engagement and thoughtful critique of rationality! Strong upvote.
While I disagree with many points I hope it gets a good rebuttal from someone (sadly I haven't the time).
I only gave a brief skim, but briefly, I think my sense about a number of the points you raise is that they are arguments that humanity's understanding of rationality is incomplete, but not that further work on it is going down the wrong track. I'm not an expert in physics, but I believe one could criticize many theories of physics for sometimes giving inaccurate predictions of phenomena in the world (e.g. dark matter, quantumn gravity), but they're still (a) able to help understand a lot of phenomena, (b) better than prior theories, and (c) further physics research seems like the right thing to do. I think I have a similar sense when you discuss ways in which humans don't appear to have static utility functions and probability distributions over world-states – treating me and others as though this is approximately true has (I believe) helped me make a lot of decisions better ("Shut up and multiply!"), and I expect that the underlying reality will end up thinking agreeing with this model in many places while substantially improving our understanding in many others.
Hey, thanks for responding! Re the physics analogy, I agree that improvements in our heuristics are a good thing:
However, perhaps you have already begun to anticipate what I will say—the benefit of heuristics is that they acknowledge (and are indeed dependent) on the presence of context. Unlike a “hard” theory, which must be applicable to all cases equally and fails in the event a single counter-example can be found, a “soft” heuristic is triggered only when the conditions are right: we do not use our “judge popular songs” heuristic when staring at a dinner menu.
It is precisely this contextual awareness that allows heuristics to evade the problems of naive probabilistic world-modelling, which lead to such inductive conclusions as the Turkey Illusion. This means that we avoid the pitfalls of treating spaghetti like a Taylor Swift song, and it also means (slightly more seriously) that we do not treat discussions with our parents like bargaining games to extract maximum expected value. Engineers and physicists employ Newton’s laws of motion not because they are universal laws, but because they are useful heuristics about how things move in our daily lives (i.e. when they are not moving at near light speed). Heuristics are what Chris Haufe called “techniques” in the last section: what we worry about is not their truthfulness, but their usefulness.
However, I disagree in that I don't think we're really moving towards some endpoint of "the underlying reality will end up agreeing with this model in many places while substantially improving our understanding in many others". Both because of the chaotic nature of the universe (which I strongly believe puts an upper bound on how well we can model systems without just doing atom by atom simulation to arbitrary precision) and because that's not how physics works in practice today. We have a pretty strong model for how macroscale physics works (General Relativity), but we willingly "drop it" for less accurate heuristics like Newtonian mechanics when it's more convenient/useful. Similarly, even if we understand the fundamentals of neuroscience completely, we may "drop it" for more heuristics driven approaches that are less absolutely accurate.
Because of this, I maintain my questioning of a general epistemic (and the attached instrumental) project for "rational living" etc.. It seems to me a better model of how we deal with things is like collecting tools for a toolbox, swapping them out for better ones as better ones come in, rather than moving towards some ideal perfect system of thinking. Perhaps that too is a form of rationalism, but at that point it's a pretty loose thing and most life philosophies can be called rationalisms of a sort...
(Note: On the other hand it seems pretty true that better heuristics are linked to better understandings of the world however they arise, so I remain strongly in support of the scientific community and the scientific endeavour. Maybe this is a self-contradiction!)
even if we understand the fundamentals of neuroscience completely, we may "drop it" for more heuristics driven approaches that are less absolutely accurate.
Because of this, I maintain my questioning of a general epistemic (and the attached instrumental) project for "rational living" etc.. It seems to me a better model of how we deal with things is like collecting tools for a toolbox, swapping them out for better ones as better ones come in, rather than moving towards some ideal perfect system of thinking. Perhaps
Relatedly, in case you haven't seen it, I'll point you to an essay by Eliezer on this distinction: Toolbox-thinking and Law-thinking.
It's definitely something I hadn't read before, so thank you. I would say to that article (on a skim) that it has clarified my thinking somewhat. I therefore question the law/toolbox dichotomy, since to me it seems that usefulness - accuracy-to-perceived reality are in fact two different axes. Thus you could imagine:
And, of course, all of these are context dependent (sometimes you may be thinking about baseballs going at lightspeed)! I guess then my position is refined into: "category 4 is great if we can get it but for most cases category 1 is probably easier/better", which seems neither pure toolbox or pure law
I'm not sure I understand how "red sky in morning, sailors take warning" can be both inaccurate and useful. Surely a heuristic for when to prepare for bad weather is useful only insofar as it is accurate?
Because it comes boxed with an inaccurate causal story.
For example, if I tell you not to lie because god is watching you and will send you to hell, this is somewhat useful because you’ll become more trustworthy if you believe it, but the claim about god and hell is false.
Maybe I am missing some previous rationalist discourse about the red sky saying. I remember reading it in books as a child, and do not know (except that it is listed here as a useful heuristic) whether it is actually true, or what the bundled incorrect causal story is. I have always interpreted it as "a red sunrise is correlated with a higher chance of storms at sea." That claim does not entail any particular causal mechanism, and it still seems to me that it must be either accurate and therefore useful, or inaccurate and therefore not useful, but it's hard to imagine how it could be inaccurate and useful.
3. Those who are more able to comprehend and use these models are therefore of a higher agency/utility and higher moral priority than those who cannot. [emphasis mine]
This (along with saying "dignity" implies "moral worth" in Death w/ Dignity post), is confusing to me. Could you give a specific example of how you'd treat differently someone who has more or less moral worth (e.g. give them more money, attention, life-saving help, etc)?
One thing I could understand from your Death w/ Dignity excerpt is he's definitely implying a metric that scores everyone, and some people will score higher on this metric than others. It's also common to want to score high on these metrics or feel emotionally bad if you don't score high on these metrics (see my post for more). This could even have utility, like having more "dignity" gets you a thumbs up from Yudowsky or have your words listened to more in this community. Is this close to what you mean at all?
I was a little confused on this section. Is this saying that human's goals and options (including options that come to mind) change depending on the environment, so rational choice theory doesn't apply?
I believe the thesis here is that game theory doesn't really apply in real life, that there are usually extra constraints or freedoms in real situations that change the payoffs.
I do think this criticism is already handled by trying to "actually win" and "trying to try"; though I've personally benefitted specifically from trying to try and David Chapman's meta-rationality post.
The idea of deference (and when to defer) isn't novel (which is fine! Novelty is another metric I'm bringing up, but not important for everything one writes to be). It's still useful to apply Bayes theorem to deference. Specifically evidence that convinces you to trust someone should imply that there's evidence that convinces you to not trust them.
This is currently all I have time for; however, my current understanding is that there is a common interpretation of Yudowsky's writings/The sequences/LW/etc that leads to an over-reliance on formal systems that will invevitably fail people. I think you had this interpretation (do correct me if I'm wrong!), and this is your "attempt to renegotiate rationalism ".
There is the common response of "if you re-read the sequences, you'll see how it actually handles all the flaws you mentioned"; however, it's still true that it's at least a failure in communication that many people consistently mis-interpret it.
Glad to hear you're synthesizing and doing pretty good now:)
Thank you for the feed back! I am of course happy for people to copy over the essay
> Is this saying that human's goals and options (including options that come to mind) change depending on the environment, so rational choice theory doesn't apply?
More or less, yes, or at least that it becomes very hard to apply it in a way that isn't either highly subjective or essentially post-hoc arguing about what you ought to have done (hidden information/hindsight being 20/20)
> This is currently all I have time for; however, my current understanding is that there is a common interpretation of Yudowsky's writings/The sequences/LW/etc that leads to an over-reliance on formal systems that will invevitably fail people. I think you had this interpretation (do correct me if I'm wrong!), and this is your "attempt to renegotiate rationalism ".
I've definitely met people who take the more humble/humility/heuristics driven approach which I outline in the essay and still call themselves rationalists. On the other hand, I have also seen a whole lot of people take it as some kind of mystic formula to organise their lives around. I guess my general argument is that rationalism should not be constructed on top of such a formal basis (cf. the section about heuristics not theories in the essay) and then "watered down" to reintroduce ideas of humility or nuance or path-dependence. And in part 2 I argue that the core principles of rationalism as I see them (without the "watering down" of time and life experience) make it easy to fall down certain dangerous pathways.
And as for the specific implications of "moral worth", here are a few:
I think copy-pasting the whole thing will make it more likely to be read! I enjoyed it and will hopefully leave a more substantial comment later.
Meta comment for LessWrong readers[1]
Something Different This Way Comes - Part 1
In which I attempt to renegotiate rationalism as a personal philosophy, and offer my alternative—Game theory is not a substitute for real life—Heuristics over theories
Introduction
This essay focuses on outlining an alternative to the ideology of rationalism. As part of this, I offer my definition of the rationalist project, my account of its problems, and my concept of a counter-paradigm for living one’s life. The second part of this essay will examine the political implications of rationalism and try to offer an alternative on a larger scale.
Defining Rationalism
To analyse rationalism, I must first define what I am analysing. Rationalism (as observed in vivo on forums like LessWrong) is a loose constellation of ideas radiating out of various intellectual traditions, amongst them Bayesian statistics, psychological decision theories, and game theory. These are then combined with concepts in sub-fields of computer science (AI and simulation modelling), economics (rational actor theory or homo economicus), politics (libertarianism), psychology (evolutionary psychology) and ethics (the utilitarianism of Peter Singer).
The broad project of rationalism aims to generalise the insights of these traditions into application at both the “wake up and make a sandwich” and the “save the world” level. Like any good tradition, it has a bunch of contradictions embedded: Some of these include intuitionism (e.g. when superforecasters talk about going with their gut) vs deterministic analysis (e.g. concepts of perfect game-players and k-level rationality). Another one is between Bayesianism (which is about updating priors about the world based on evidence received, generally without making any causal assumptions) vs systemisation (which is about creating causal models/higher level representations of real life situations to understand them better). In discussing this general state of rhetorical confusion I am preceded by Philip Agre’s Towards a Critical Technical Practice, which is AI specific but still quite instructive.
The broader rationalist community (especially online) includes all sorts of subcultures but generally there are in group norms that promote certain technical argot (“priors”, “updating“), certain attitudes towards classes of entities (“blank faces“/bureaucrats/NPCs/the woke mob etc), and certain general ideas about how to solve “wicked problems” like governance or education. There is some overlap with online conservatives, libertarians, and the far-right. There is a similar overlap with general liberal technocratic belief systems, generally through a belief in meritocracy and policy solutions founded on scientific or technological principles.
At the root of this complex constellation there seems to be a bucket of common values which are vaguely expressed as follows:
There is also a fourth law which can be constructed from the second and third: By thinking about this at all, by starting to consciously play the game of thought-optimisation and higher order world-modelling, you (the future rationalist) have elevated yourself above the “0-level” player who does not think about such problems and naively pursues their goals.
It is easy to suggest that I am strawmanning the opposition, so I will try to align my statements with quotes from the community I am analysing. A more direct formulation of the first two principles can be found in the following article [2]. Rationality as formulated by Eliezer Yudkowsky contains two core ideas which are also summarised into short and catchy phrases:
Here, epistemic rationality roughly corresponds to my first proposed core belief, and instrumental rationality the second. As to the third principle, while Yudkowsky explicitly notes that you can win without “winning at others” expense” the word “winning” still suggests a triumph against some opposition in a game-like situation. This focus on games will return later, but for now it is enough to note that to “win” is explicitly correlated here with agency and utility for your chosen values or causes, hence validating my third core belief. The fourth belief is slightly more nebulous, but it rears its head when the melodramatic or quasi-satirical side of rationalism emerges:
Here, to be dignified is to be a rationalist who sees the “hard scary real problems” as opposed to those who delude themselves by working on “easy irrelevant fake problems”. It’s not difficult to extend the implication from “dignity” to “moral worth”, especially given the tone of this article.
Rationalism is path-dependent
Eliezer explicitly calls out probability and decision theory in his definitional article above, and writes the following about probability theory: “It’s all the same problem of how to process the evidence and observations to update one’s beliefs. Similarly, decision theory is the set of laws underlying rational action, and is equally applicable regardless of what one’s goals and available options are. [Emphasis mine]”
We will for a moment set down probability and focus first on decision theory, though I argue what I am saying applies in general to them both. Eliezer lays down two important qualifiers when discussing rational action—First, that agents may want different things and therefore have different goals. Second, he notes that agents may have different options available to them to achieve said goals. These are true and worthy constraints. However, I argue that some linguistic sleight of hand has occurred here. Leaving aside any ontological arguments of whether humans truly “want” anything, or only appear to want things etc., the landscape of choice includes not only where you want to go and how you might get there, but also the terrain you will walk on, your environment and context. Humans are not static containers of properties like “wants X” or “fears Y”, but dynamic systems constantly updating from their environment[3].
At first, my focus on context seems like a distinction that can be easily folded into either one’s goals or one’s options, but I would suggest that this is not easily done. While a rationalist view posits life as the pursuit of long term goals that remain like north stars, most of our short term goals come from what we see as our context: As this article about business cash flow makes clear, the environment we think we are in (what he calls the axioms or principles we set up) determines what we see as optimal strategies far more than we think—if you think your job is to make profits, your “game” looks a lot different from Jeff Bezos, who sees business in the context of cash flows and financial optionality[4].
Similarly, our options too are often defined not by our ability to reason through what the optimal path is, but rather by what our environment makes available to us. Being trapped on a sinking ship with no lifeboats limits your options even if you are a world-class reasoning expert. Finally, how we approach deciding at all is context-dependent: No matter how hard we try to reason ourselves to the Cartesian root of cogito ergo sum, we are brought up in different environments and exposed to different ideas that lead us to be “initialised” in random ways, like the starting weights and biases of a neural network. Often such biases are not fully known even to us, but they can affect our judgement profoundly: try managing a social crisis while hungry, or suffering from a mild headache, or in a quiet library. Our “rational decision” should not be described as a cognitive function we perform on our immutable goals and the actions available to us. Instead, it is a largely subjective thought process we perform based on our environment as we observe and update our internal world-model. All rationality is, to some extent, bounded rationality.
It is this context-dependence that makes applying rational choice theory to the real world exceptionally difficult. Indeed, one of the hallmarks of a certain type of hypothetical malicious AI is that it cannot take into account its context, endlessly optimising for an abstract objective without considering whether that pursuit might eventually harm itself or its creators. As Nassim Taleb writes in his book Antifragile:
In the example, we see that what holds true for a certain context (the game theoretic bargainer) does not hold true in a different one, and in fact fails to achieve the instrumental outcome of rationalism (in this case, “winning” the bargain in a way that makes the bargainers satisfied). Rational choice theory takes as its starting point the perfect agent in a perfect void, and therefore matches the experience only of game theorists role-playing as hypothetical people standing before a rapidly approaching trolley. While it is true that two identical clones in two identical universes faced with the same decision at the same moment in time would make the same decision if they were perfectly rational agents, the same cannot be guaranteed for any other situation, especially in our messy and chaotic world[5]. From our earliest moments we are randomly initialised, our first subconscious priors determined by events outside of our control, our organically developed preferences and these subconscious priors influencing how we take in any new information afterwards. Thus I argue that, in the language of rationalism itself, rationalism is path-dependent.
Games and Game Theory
Perhaps because applying their theories directly to the real world is difficult, it is common for various social, political and economic situations to be described in formalised forms as games by rationalists [6]. I argue that a game-theoretic framing of such problems is counterproductive and destructive to meaningful resolutions of such problems. To begin our analysis, however, we need a concrete definition of what a game is and why games matter:
In extract above, Nguyen proposes that the main artistic medium of games is agency: while games incorporate other art forms in their production, the true defining aspect of a game is the cultivation of a possibility space in the mind of the player. The agency is imagined because it is not true agency, in the sense that the player’s choices do not impact the world outside of the game. The goals of the game (getting the ball into the net, reaching a high score in a certain time, defeating the boss) are illusory, but the player enters into a magic circle or contract to take them seriously for the duration of the game, achieving these goals within the rules of the game and deriving satisfaction from that triumph. Taking the imaginary game-goals too seriously and acting outside the set parameters to achieve them (i.e. “cheating”) is deemed a false victory because it is a sign that you are performing a Goodhart’s law-style degenerate optimisation: For most people if you use a rocket launcher to shoot the football into the goal you won’t score any points, because you are no longer meaningfully playing in the agentic space set out by football’s designers. In other words, you’re no longer playing football, so the points don’t matter.
With this analysis in mind, we can take a look at whether the prisoner’s dilemma is, indeed, a game. We can extend our previous insight about rationalism to these games as well, considering not only the goal and the choices at hand but also the constraint or contexts implied or communicated through the game’s rules. Therefore, the primary elements of our analysis are whether there is some cultivated agency or possibility space set out, whether this agency is bounded by a set context, and whether those moves are tied to achieving some goal that is game-dependent and hence illusory. Under this analysis, the prisoner’s dilemma is a game:
So long as all of these conditions are fulfilled, a game-theoretic analysis can be applied: for any given option, analyse the expected payout for each player, and assess each combination of moves and payouts to derive a Nash equilibrium. The study of game theory can therefore be described as the study of how to most effectively exercise a player’s in-game agency. Indeed, it is this exercise that powers some of game theory’s most unintuitive and impactful results—The standard analysis of the prisoner’s dilemma says that, rather than cooperating to achieve a better outcome for everyone, if each prisoner is to maximise their objective they must defect. A more general case of this analysis is known as the tragedy of the commons, where many players can choose to cooperate or defect over a period of time in the context of managing a shared good all players have access to.
Now that we have defined a game, we can see why solutions to the prisoner’s dilemma that take the form “the prisoners should just break out of prison”, “the prisoners should have a secret code” etc. are unsatisfying from a game-theoretic perspective: it’s like saying the most effective way to win at football is to bring a drone and a rocket launcher. Similarly, games like the Trolley problem are interesting or frustrating (depending on your point of view) thanks to the constrained agency of the player: arguing that the setup of the problem is contrived is like saying that the player ought to be able to pick up the football since they have hands. In some sense, all games are contrived, the difference is the satisfaction we gain from obeying their contrivances.
Come on, this is serious
Unfortunately, there is a more insidious aspect to game theory. The prisoner’s dilemma or the tragedy of the commons is often used as an argument for why cooperation is impossible or often fails in the real world. Moved by such arguments and confirmation bias with real life examples, defecting becomes accepted as a baseline policy, and any cooperation is seen as a miracle or the result of some edge case such as a higher power imposing cooperation on two squabbling players. On the other hand, such a pessimistic belief is only insidious if it is derived from false premises: after all, if rational or rational-enough actors truly have a foolproof analysis as to why defection is the best base policy in cooperation games, then this would be a a tragic truth rather than a dangerous lie.
Why might this projection of conclusions from games to reality be incorrect, then? Game theory purports to be about perfect agents and their choices, so it is easy think of it as an ideal form of decision-making to strive towards. However, recall that the definition of a game is not only what you can do, but what you cannot. As we discussed earlier with theories of rational choice, games and optimal strategies are contextually dependent. Indeed, games explicitly inject their context as the axiomatic foundations of the game-world in the form of rules and premises. The possibility space of most games constrains at least as much as it enables: If you are able to break these strict constraints on player agency then the conclusions of any game-theoretic analysis fall apart, their payout matrices crumbling into the mire of relativity and infinite hypotheticals. What’s the correct play for the goalkeeper when your opponent has a rocket launcher?
We can of course respect this artificiality to play a game and extract satisfaction, e.g. by agreeing not to pick up the ball in football. Such self-awareness does not go both ways, however. The implication of many game-theoretic analyses of geopolitical cooperation, climate cooperation etc is that if rational players defect in the game they have set up they will also defect in real life, that the artificial model of the game is an accurate enough model of reality such that conclusions in one can be effectively projected to the other. In other words, this brand of analysis demands that you keep real life non-linear problem-solving out of the game, but demands that the conclusions drawn from the game be applied into real life with no such caveats. The game-theoretic football analyst knows that that touching a ball with your hands is possible only for the goalkeepers of the world, and even then only if they are standing in front of goal-shaped objects.
The usual game theoretic solution to such arguments is to shape the constraints of the game to better model reality. We can play many rounds of the prisoner’s dilemma, with players having some memory of previous rounds, such that their agency can be better shaped by their models of their opponent. We can introduce means of signalling intent into the commons game, or introduce more parallel tracks in the trolley problem[7]. Yet, no matter how elaborate these premises become, they remain games. Recall that participating in a game involves the conscious adoption of illusory game-goals as your temporary objectives. Standing in two rows before the referee blows the whistle in football is a form of conscious buy in, the same thing we do when we read the rules of the trolley problem and agree to consider what we would do in that game-world seriously. The necessary pre-condition of playing a game is to know that a game exists and to agree to play. In most social situations, this tabula rasa-eque buy in does not exist, both because of contrasting factors like honour, emotion, pre-commitments etc. but also because of imperfect communication and information asymmetry. Kicking a ball at someone on their phone does not mean that you are now playing football with them.
Of course, game theory does have particular applications. Game theory describes useful equilibria that can be aimed for or avoided in the design of large scale social systems like markets or incentive structures, where agents are relatively impersonal, have access to ample information and consciously buy in to participation. However, in the domain which game theory finds itself (mis)applied most often, that of the social sphere, knowledge of such a theory can often become counterproductive. This is because the knowledge of some optimal strategy in the context of a game biases one’s attitudes towards both what actions they should take and even what actions are available in their internal conception of the “game”. In short, the reference frame of “I am playing a game” causes them to rule out cooperative or supposedly “suboptimal” strategies. Game theory, designed to assist in reasoning about problems becomes a hazard to reasoning accurately about problems because it turns problems into games and imagines that conclusions can propagate backwards with perfect accuracy. The commonsensical formulation of this conclusion is again Taleb’s:
Models Collapse in Contact with Reality
So far, what I have described are instrumental failings of rationalism; in general I am attempting to attack the second core belief that rationalism makes us better at solving problems or achieving our goals. I will now attempt to question the first core belief, that rationalism helps us understand the world. To do this, I will introduce the concepts of irreducible and fractal complexity.
For the sake of rigor, I will begin with questioning the idea of a complete and total high level model of society through some light application of chaos theory: After all, if such a model is possible, all else follows for the project of epistemic rationality. The field of chaos theory is dominated in popular culture by the idea of the butterfly effect. However, an idea I find more interesting is the concept of the saddle point, a form of metastable equilibrium (temporary resting point) where any slight disturbance can cause the future paths of the object at equilibrium to diverge wildly. It is these features that make chaotic systems “highly sensitive to initial conditions”, and they are present in theoretically deterministic systems like the weather, the stock market, etc. In fact, since self-contained cyclical chaotic systems with strange attractors return to such metastable saddle points regularly, they are regularly highly sensitive to their conditions, a feature that makes predicting the future for such systems beyond extremely near-range forecasts nearly impossible[8].
Before you protest that such features are only true of complex physical systems, I will point out that our societies are also composed of billions of complex components in a complex and ever-shifting environment. They also seem to oscillate between periods of relative predictability (“peace”) and periods of extreme instability, where one errant action or bullet can change the fate of millions (“war“), with both seemingly inevitable in retrospect but somehow always escaping our best predictive efforts until it is too late[9]. We can combine the idea of saddle points with Taleb’s idea of black swans, unpredicted events with high negative or positive impact on the state of the complex system as a whole[10]. This can give us a rough sense of “irreducible complexity”, the idea that higher order models of complex and chaotic systems can and must spiral into divergent and irreconcilable predictions beyond the near range. This applies to human simulators as well as computer simulators: Predicting the 2024 US election is one thing, predicting the 2032 election is another entirely—after all, who can guarantee that there will even be a 2032 US election, and if there is that the same parties will participate?[11]
If a total top down model is difficult to achieve, what about precise models of smaller sub-systems within society, like nations or corporations? Here is where the idea of fractal complexity comes in. While the actions of, say, a large corporation like Facebook seem monolithic, ruthless, or even to some degree rational (remember again that in a subjective path-dependent world the definition of rationalism is post-hoc), whenever we look deeper we find these systems composed of innumerable, equally complex sub-systems. Down to the level of individuals (as one will find with any biography of a notable historical figure), accurately accounting for the actions of the various parts of a marketing department, or a mind, or a military unit is an incredibly complex endeavour, and any model becomes a game with its own baked-in presumptions.
That is not to say that heuristic rules for properties of complex phenomenae do not exist. For example, we can model the pressure in a container using the ideal gas law without massively expensive simulations of millions of atoms. However, the success of classical physics in this regard relies on simplifications that come from a difference in scale between atoms and coke cans, something we simply don’t have access to in the social sphere. Furthermore, the mechanisms they model are non-agentic, and therefore largely exhibit Brownian (random) motion. People are not so simple: If you disagree, I refer you again to the stochastic nature of the stock market, where every incentive is there to refine the science of prediction and we’ve still gotten nowhere without some kind of insider edge.[12]
Probability and His Problems
Okay, okay, okay, I hear you say. But surely probability isn’t wrong? After all, this is what the basic program of epistemic rationality is:
There are two problems with this account of epistemic rationality and Science (capital-S). The first and most obvious is that “get up and make a sandwich” level Bayesian statistics[13] translate poorly to modelling outcomes of complex distributions in society, nature, politics and economics. Just because the causal chain between “it is raining” and “the floor will be wet later” or “my shoelaces are untied” and “I see that my shoelaces are untied” is obvious does not mean that “Federal Reserve interest rates are raised” and “consumer purchasing power goes up/down/sideways” is obvious.
The second and perhaps more dangerous error is that this is not how science operates. Yudkowsky says that science is “reflective reasoning about a more reliable process for making the contents of your mind mirror the contents of the world”, and carried out by “performing replicable experiments to falsify theories”. However, when we learn physics we do not start by exhaustively verifying the laws of motion, then the laws of thermodynamics, then move on to recreating the experiments that proved the existence of atoms, quarks, the Higgs Boson or relativity. We may get tasters of such scientific experiments, to “get a feel” for the scientific method, but what we do a lot instead is take other people at their word. Some of these people happen to be called Niels Bohr and Albert Einstein, and we are told that there are very good reasons for taking them at their word, but taking someone’s word for it is just that—taking someone’s word for it[14]
But why is science effective, then? Why don’t we take the words of flat earthers, or moon landing conspiracists, or Christian scientists? If science is truly about “making the contents of your mind mirror the contents of the world”, what business have we learning and using erroneous theories like Newtonian motion? Yudkowsky betrays a fundamental misunderstanding of science when he writes in awe about “the potential power [science] bestows on us as individuals, not just scientific societies”. Because it is precisely the other way around: we overestimate the power science gives the lone scientist, and underestimate the role played by the scientific society. Alone, we are prey to confirmation bias and cognitive dissonance and a thousand other tricks our brain plays on us. The most effective way to resolve these errors is not to sit still and think very hard, but to submit our ideas to others for replication, falsification, and criticism, even when our opponents disagree with us strongly. Over time, ideas are adopted as knowledge and become taken as granted, the foundations of further research, making it no longer necessary for us to spend hours splitting the atom in every undergraduate physics class. Alone, what we have is ideas, together, what we have is knowledge:
So far, I seem to have launched a full scale attack on the very idea that we can create universally applicable higher order representations of the world[15]. Despite this, it seems intuitive that we should be able to know some things about the world. How does this work then?
Heuristics over Theories
In this section I introduce the idea that (local) heuristics may be superior to (universal) theories in terms of predictive power and practical utility. To be clear, when I say a theory or model I mean something on the order of Einstein’s general theory of relativity. Heuristics are general estimates like “will I have time to cross the road before that car hits me” or “is this book going to be a good read for me”. More mathematical heuristics might be “should I try solving this problem with induction” or “what probability distribution fits this data best”?
At first, heuristics seem like inferior, degraded forms of theories: where theories are universal and rigorous, heuristics are situationally dependent and fuzzy; where theories are precise and elegant, heuristics are rough and often extremely path-dependent. Heuristics are borne out of personal experience and practice while theories are precisely captured in mathematical formulations and can be shared across many different contexts. Compare the instincts of a New York options trader with a neophyte relying on the Black-Scholes-Merton formula for options pricing.[16]
However, perhaps you have already begun to anticipate what I will say—the benefit of heuristics is that they acknowledge (and are indeed dependent) on the presence of context. Unlike a “hard” theory, which must be applicable to all cases equally and fails in the event a single counter-example can be found, a “soft” heuristic is triggered only when the conditions are right: we do not use our “judge popular songs” heuristic when staring at a dinner menu.
It is precisely this contextual awareness that allows heuristics to evade the problems of naive probabilistic world-modelling, which lead to such inductive conclusions as the Turkey Illusion. This means that we avoid the pitfalls of treating spaghetti like a Taylor Swift song, and it also means (slightly more seriously) that we do not treat discussions with our parents like bargaining games to extract maximum expected value. Engineers and physicists employ Newton’s laws of motion not because they are universal laws, but because they are useful heuristics about how things move in our daily lives (i.e. when they are not moving at near light speed). Heuristics are what Chris Haufe called “techniques” in the last section: what we worry about is not their truthfulness, but their usefulness.
Moreover, it may be that, as much as we find heuristics distasteful, we have to work with them anyways. Heuristics work at the subconscious level, as tacit and internalised knowledge rather than cumbersome and externalised knowledge. This means that, whether we like it or not, heuristics guide our actions whenever we don’t stop to explicitly reason out our options. When I’m deciding what to eat for dinner or what to say next at a party or how to steer my bike, I’m using heuristics. Improving our decision making, therefore, is inextricably linked with improving our heuristics.[17]
More concretely, shifting the balance of our thought from applying perfect theories to cultivating and cataloguing useful heuristics allows us to make meaningful use of, rather than suppress, those instinctual and subconscious parts of our brain we call hunches, biases, and emotions. Properly “trained” by experience, these systems can allow us to generate spontaneous connections between ideas, recall precisely the correct answer to a question we had no idea how to solve, or recognise danger before it arises, and they should not be derided or ignored. Rationalism, as a whole, privileges working things out in the conscious brain to trusting our gut, and that is I argue one of the reasons that, when true emergencies appear and real difficult decisions are on the line, it is rare to see Pareto-optimal strategies being employed.
A Heuristic for Life: Conscious reinvestment
Suppose, then, that you accept in principle that heuristics might be a better way for handling problems than having a perfect decision theory, both because we are inherently only capable of limited cognition and because heuristics imply a recognition and respect for context. How does that play out over a lifetime, how does this statement cash out?
Much is made in rationalist circles of the need to expand your optionality, to accrue options and choices and agency by recognising the choices at hand. Of course, in an abstract environment having more policies available to achieve your goals is better than having less, especially if you can keep all options open while constantly getting more. Much effort is also expended in breaking down the artificial value and goal structures imposed by society, with the general belief that people should be free to determine their own goals rather than bowing to social pressure. However, in real life these liberations can manifest as a kind of malaise, a sense that you could do everything and therefore cannot easily decide on doing anything. It can feel like we’re standing in a great desert, with options stretching out all around us in a combinatoric explosion, with nothing to do except to walk endlessly down one decision tree or another.
How can we achieve what we want when the options are so vast and opaque, when we have liberated ourselves from society’s carefully manicured pre-set routes? When love and fear and hope and democracy are just so many signalling games we play, what does it even mean to live a good life?
The first thing we can do is follow the “north star” approach, where we fix some point in the sky as our ultimate destination and walk unceasingly forward. This naturally runs into problems of degenerate optimisation (which can in many cases sunder your chances of actually achieving your goal, especially in a social context), but it also invites a painful and unsatisfying form of life in which there is no way to embrace unexpected opportunities or change your mind without giving up your entire life project and branding yourself “irrational”. Instead, we might try adopting the heuristic approach to achieving a goal: define a heuristic direction pointing where we roughly wish to go, and then taking the option to move in that direction when the situation seems appropriate, rather than at every possible opportunity. By “loosening our grip” on our agency, we allow ourselves the freedom to experiment and in turn gain more information about our goal, iteratively improving our ability to recognise and exercise the options our circumstances afford to us.
But what if you cannot be sure what your true preferences are? Then it may be necessary to perform a second-order heuristic search. Since you are reading this, you have already been “randomly initialised”, with a set of opportunities and people around you that you may like more or less or understand more or less. Then you can begin your heuristic search: When the opportunity arises, lean in to investing in people and causes rather than not, but don’t hold any idea or person particularly strongly at the start[18]. Over time, reinvest in things you find more meaningful, and exit situations that don’t give you meaning (but not too hastily, hindsight is a powerful introspection tool). Once you have a clear sense of what appeals to you and what is meaningful to you, you can start on the broader project of heuristically pursuing meaning where it leads you.
In high school, I was lonely, loveless, and miserable, on the verge of slipping into the right-wing radicalisation pipeline. I resolved to change my situation with no clear idea of how to do so. In university, I developed this method out of desperation, throwing myself into every group and relationship I could find in COVID to give my life some kind of attachment and meaning. Over many painful experiences, I got a rough grip of what I actually wanted to learn about and do in the world. This was not a clean, optimal path, but precisely because it was not optimal I managed to gain lessons and friends from unexpected places, pushing back my unknown unknowns and refining my approach to life[19]. This method is what I call conscious reinvestment, and it partially saved my life. I hope it helps you.
This is part of a series of posts/thoughts I've put together as I come of age in the 2020s. Perhaps particularly relevant is the essay I've linked on some of the feelings I have about rationalism, but the whole collection might have material that is generally of interest (incl. my thoughts on why the current AI/LLM development paradigm is harmful and my proposed solution, including prototype code that runs).
To be clear: I'm not coming at this from the perspective of "outsider sees thing bad, points and laughs" - I have read all of HPMOR and been variously active in stuff like AI safety, but it does represent some of the points of tension I have felt come up while in conversation with rationalists/those who are rationalist-adjacent. I hope, if nothing else, there is something of interest to consider here.
Eliezer Yudkowsky, What Do We Mean By “Rationality”? in https://www.lesswrong.com/posts/RcZCwxFiZzE6X7nsv/what-do-we-mean-by-rationality-1
For a much longer exploration of these ideas, see “Context changes everything” by Alicia Juarrero: https://mitpress.mit.edu/9780262545662/context-changes-everything/
And, of course, the same is true the other way around: witness the crypto scams that boast high EBITDAs but do not realise that their cash flows are not materialising. Delusions about your environment are often more dangerous than ignored facts about your environment.
The standard rationalist counterargument to this line of reasoning appears to be that, while it is indeed difficult to be perfectly rational, one can still strive to be more rational overall in the situations one finds themselves in. Of course, such a judgement is itself subjective and based on one’s assessment of the environment around them, the energy they have to evaluate the expected value of their actions, etc. If my goals, my actions to achieve those goals, and also my assessment of how efficient I have been at achieving my goals are all subjective, then almost any combination of goal and action can be made to appear “rational” by post-hoc reasoning. Hence, the meme of the 4-dimensional chess player for whom every failure is just another step towards total domination.
I would argue that the Effective Altruism community has a similar problem with formulating different interventions as game-like policy trade offs, as if they are triaging patients at a hospital.
A second, easier strategy is to lean on our natural ability to recognise patterns and say that while the game isn’t a perfect match for real life, the lessons of the game should carry over nonetheless. As I hope I have explained sufficiently, the mathematical validity of game-theoretic analyses do not survive such creative re-interpretations. On the other hand, game designers who act with a clear knowledge of their context and the agencies they are cultivating can create quite beautiful metaphors of real life that don’t rely on mathematical conclusions to deliver their lessons. Cole Wehrle discusses this in the context of wargaming here: https://www.youtube.com/watch?v=wi_MpZxLPFM
For some humorous evidence of the difficulty of predicting chaotic systems, see the following article about how monkeys beat the stock market: https://www.forbes.com/sites/rickferri/2012/12/20/any-monkey-can-beat-the-market/
A general sense of such cyclical alternation between moments of relative predictability and total uncertainty was not unknown in the past. The venerable Chinese novel The Romance of the Three Kingdoms begins as follows: “Of the great matters under heaven, that which is long divided must unite, and that which is long united must divide”.
Unpredicted because they are exceptional and “break the historical record”, and therefore cannot possibly feature in risk analyses that only look at historical worst cases; but also because they are often literally out of the distribution of conceivable events.
It would not be amiss to call forecasting a game using the scheme we have established above.
Here’s another way to think about it: if a system does not contain a large amount of random actors but rather a large amount of actors in a temporary metastable (read: moderate) state who can fall into more stable (read: radical) states and influence others thereby, a small difference in which actors change state first will have a contagious effect and therefore lead to high divergences in the final predictions.
A general example being “I have five drawers, I know there’s a 90% chance my socks are in one of the drawers, and have opened two drawers, what are the odds it’s in the third?”
Before the usual counter-claims about evidentiary standards and falsifiability are raised, I will note that Jan Hendrik Schön got very far in physics by literally making up his numbers and asking people to take his word for it: See BobbyBroccoli’s excellent series on the matter at https://www.youtube.com/watch?v=nfDoml-Db64 . At the end of the series it is noted (by a scientist who was critical of him, no less) that science requires a great deal of trust, and that exhaustive requirements for every experiment to be repeated and verified would almost certainly grind scientific research to a halt.
Some, like Taleb, even claim that the delusion that we can construct such theories comes from the success of heuristics, rather than people deriving heuristics from sound theories.
Taleb and Haug, “Option traders use (very) sophisticated heuristics, never the Black–Scholes–Merton formula”, in the Journal of Economic Behavior and Organization
This is, apparently, more or less what people mean when they say that they are improving their forecasting ability. I argue that such a valuable skill is wasted at “horse race“-focused betting games (after all, it seems unlikely that superforecasters will branch into weather forecasting any time soon).
For me, this was almost impossible, but you learn to loosen up somewhat after a few heartbreaks.
Incidentally, the episode that inspired me to formalise this method was coming across Venkatesh Rao’s psychic assault upon all I believed in, also known as his dissection of American sitcom The Office. See here: https://www.ribbonfarm.com/2009/10/07/the-gervais-principle-or-the-office-according-to-the-office/ . As an exercise for the reader, I invite you to re-analyse this extremely well written and persuasive theory focusing instead on psychological pressures exerted by the office context, which Rao entirely ignores.