I claim that this series of posts will be a decent training regime for applied rationality. Accordingly, I think that one of the first steps is to tell you what I even think applied rationality is. However, since "applied rationality" is a little hard to describe succinctly, I will describe it through example and metaphor. To make the process of triangulation easier, I will also provide three such descriptions.
Disclaimer: I adapted the first 2 takes from CFAR; the third is my own.
In this post-enlightenment era, we have this thing called science. Loosely speaking, science is a set of norms that allow for the gradual development of true and verifiable bodies of knowledge. Some examples of these norms include: testing hypotheses, peer review, and discarding theories that have large amounts of evidence piled against them. A short description of these norms is that whenever someone has a question about how the world works, the proper answer is “why don’t we just check.”
In my view, however, science has (at least) two major flaws.
Science is slow. The norms of science are sufficient to eject false knowledge from scientific canon eventually. They are most certainly not sufficient to eject false knowledge quickly. As the replication crisis demonstrates, scientific canon has contained, for many years, large swaths of false knowledge. Theories that are false are discarded in the limit, but they are not discarded as soon as a competing theory has sufficient evidence. Sometimes theories have the same consequences, so “just checking” can’t distinguish between them.
Science is expensive. There are many resources that science must consume to generate knowledge. Foremost among these resources are time and money, but it also sometimes requires certain amount of status. In general, science can only make broad prescriptions about how any given individual engages in the world. Of course, you can do science on yourself, but it also costs a fair amount of resources. Sometimes “just checking” is infeasible to do at such a small scale.
I claim that applied rationality does a pretty good job of filling both these niches. If you're practiced at the art of arriving at true beliefs, I claim that you can change your mind in as little as 5 minutes, upper-bounded by perhaps a single year. On the other hand, science routinely takes more than a single year to change its mind about things like "power poses".
Similarly, when making decisions on a day-to-day basis, it is difficult to do proper experiments to determine which decisions to make. Science is a powerful and expensive tool -- one that applied rationality can tell you when to use. When you find science insufficient for the task, applied rationality can help you make good decisions using information you already have.
Compressed into a single sentence, applied rationality fills the gaps of science in the pursuit of truth.
Imagine a monarch who rules an entire kingdom. This monarch is tasked with making decisions that benefit the kingdom. This monarch is not stupid, so they surround themself with many expert advisors. For example, the monarch might have economic, political, environmental, interpersonal, and legal advisors. However, the advisors are not quite properly restrained, so the monarch finds that their advisors do not quite know the bounds of their own advice.
Sometimes, it is clear which advisor the monarch should listen to: when deciding how much money to print, the monarch should probably listen primarily to the economic advisors. However, the printing of money is bound by law, so the legal advisors should probably be consulted. Printing money is also a political action, so the political advisors should also be consulted. Additionally, any movement in the economy has environmental ramifications, so the environmental advisors might also be consulted.
As we see, even in a seemingly clear-cut case, there are reasons for consulting all of the advisors. Now imagine a decision in a domain for which the monarch has no explicit advisors. In this scenario, it seems to me like the vast majority of the work in making a good decision is to decide which advisors to listen to.
Obviously, this is all a metaphor. To make it explicitly clear, you are the monarch. The kingdom is everything that you value. The advisors are all the sources of information you have available to you. These sources might include actual advisors, like your friends or people you hire, but it also includes things like books, the internet, your explicit reasoning, your intuition, your emotional reactions, your reflective equilibrium, etc. Crucially, you have situationally bad advisors. When there is a tiger running at you at full speed, it is vital that you don’t consult your explicit reasoning advisor.
I claim that similarly to the imaginary monarch making decisions, most of the work that goes into making good decisions is choosing which sources of information to listen to. This problem is complicated by the fact that some sources of information are easier to query than others, but this problem is surmountable.
Compressed into a single sentence, applied rationality is the skill of being able to select the proper sources of information during decision-making.
Imagine that you’re a human being with values. You are in some environment and have certain actions available to you that manipulate the environment to better satisfy your values. Clearly, you want to take the action that makes the world score higher according to your values. You’ve heard of this thing call “rationality” and decide that the most “rational” policy is to take the action that maximizes the expectation of value.
You’re faced with a decision as to what to eat for dinner. You look at all the possible options and you figure out distributions for how much value you’ll get from all the options. Then you take the expectation of the value under all the distributions. By the time you have finished, the restaurant has closed and you’re still hungry. You decide that “rationality” is bunk and you should go with your intuition in the future.
This example might seem a bit contrived (and it is), but the general principle still holds. In this scenario (as in real life), you’re a human being with values. This means that you are unfortunately bounded along nearly all dimensions (for now). In particular, you have bounded memory, a bounded action space, and bounded compute. Any “rationality” that doesn’t have a way of dealing with boundedness doesn’t seem like a very good “rationality.” This type of “rationality” is not what I mean by applied rationality.
Julia Galef once defined eudaimonia as “happiness minus whatever philosophical objections that you have to happiness.” In a similar vein, I define applied rationality as “rationality minus whatever reasons you have for why rationality will never work.”
If you think that being “rational” doesn’t work because you have to consider emotional reactions to things, then what I mean by applied rationality is a rationality that takes emotional reactions into account.
If you think that being “rational” doesn’t work because humans need to take breaks sometimes and do what they want, then what I mean by applied rationality is a rationality that allows you to take breaks and do whatever you want.
If you think that being “rational” doesn’t work because sometimes you need to use intuition, then what I mean by applied rationality is a rationality that allows you to use intuition.
I think you get the point by now.
Compressed into a single sentence, applied rationality is a system of heuristics/techniques/tricks/tools that helps you increase your values, with no particular restriction on what the heuristics/techniques/tricks/tools are allowed to be.
I have described to you three perspectives on what applied rationality is. An exercise for the engaged reader is to find a friend and explain to them what applied rationality is to you. I encourage you to make use of metaphor and analogy, which are, in my experience, the strongest tools for quality explanation. I also want to hear what your explanation is, so I would enjoy it if you commented.