In my opinion, "a rationalism" (IE, a set of memes designed for an intellectual community focused on the topic of clear thinking itself) requires a few components to work.
The sequences told a story in which the first role was mostly played by a form of Bayesianism, and the second was mostly played by the heuristics and biases literature. The LessWrong memeplex has evolved somewhat over time, including forming some distinct subdivisions with slightly different answers to those two questions.
Most notably, I think CFAR has changed its ideas about these two components quite a bit. One version I heard once: the sequences might give you the impression that people are overall pretty bad at Bayesian reasoning, and the best way to become more rational is to specifically de-bias yourself by training Bayesian reasoning and un-training all the known biases or coming up with ways to compensate for them. Initially, this was the vision of CFAR as well. But what CFAR found was that humans are actually really really good at Bayesian reasoning, when other psychological factors are not getting in the way. So CFAR pivoted to a model more focused on removing blockers rather than increasing basic reasoning skills.
Note that this is a different answer to the second question, but keeps Bayesianism as the overarching theory of rationality. (Also keep in mind that this is, quite probably, a pretty bad summary of how views have changed since the beginning of CFAR.)
Eliezer has now written Inadequate Equilibria, which offers a significantly different version of the second component. I could understand starting there and getting an impression of what's important about rationalism which is quite distant from Bayesianism: there, the primary story about rationality is social blockers, to which the primary antidote is thinking for yourself rather than going with the crowd. Why is Bayesianism important for that? Well, the answer is that Bayesianism offers a nuts-and-bolts theory of how to think. You need some such theory in order to ground attempts at self-improvement (otherwise you run the risk of making haphazard changes without any standard by which to judge whether you are thinking better/worse). But the quality of the theory has a significant bearing on how well the self-improvement will turn out!
I think it's important that the overarching reasoning was some form of probabilism for "obvious" reasons I won't go into.
I think it was important that it was Bayesianism in particular for a few reasons, some better than others.
But what CFAR found was that humans are actually really really good at Bayesian reasoning, when other psychological factors are not getting in the way.
What's the source for that claim? Is that a public position of CFAR?
See Eliezer's post Beautiful Probability, and Yvain on 'probabilism'; there's a core disagreement about what sort of knowledge is possible, and unless you're thinking about things in Bayesian terms, you will get hopelessly confused.
Also, Yvain's article really helped clarify how Bayesianism is important, although maybe I'd call it more probabilistic reasoning more than anything else
Seems to me that there is a disagreement on how the "probabilistic reasoning" is done correctly, and Bayesianism is one of the possible answers.
The other is frequentism, which could be simplified (strawmanned?) as "the situation must happen many times, and then 'probability' is the frequency of this specific outcome given the situation". Which is nice if the situation indeed happens often, but kinda useless if the situation happens very rarely, or in extreme case, this is the first time it happened. In those cases, frequentism still provides a few tricks, but there doesn't seem to be a coherent story behind them, and I think that in a few cases different tricks provide different answers, with no standard way to choose.
More technically, Bayesianism admits that one always starts with some "prior belief", and then only "updates" on it based on evidence. Which of course invites the questioning of how the prior belief was obtained -- and this is outside the scope of Bayesianism. However, many frequentist tricks can be interpreted as making a Bayesian update upon an unspoken prior belief (for example, a belief that all unknown probab...
Haven't read the book, so I looked at some reviews, and... it seems to me that there two different questions:
a) Are there universal laws in math and physics? (Yes.)
b) Are the consequences of such laws trivial? (No.)
So we seem to have two groups of people talking past each other, when one group says "there is a unifying principle behind this all, it's not just an arbitrary hodgepodge of tricks all the way down", and the other group says "but calculating everything from the first principles is difficult, often impossible, and my tricks work, so what's your problem".
To simplify it a lot, it's like one person saying "multiplying by 10 is really simple, you just add a zero to the end" and another person says "the laws of multiplication are the same for all numbers, 10 is not a separate magisterium". Both of them are right. It is very useful to be able to multiply by 10 quickly. But if your students start to believe that multiplication by 10 follows separate laws of math, something is seriously wrong. (Especially if they sometimes happen to apply the rule like this: "2.0 × 10 = 2.00". At that moment you should rea...
I think "probabilistic reasoning" doesn't quite point at the thing; it's about what type signature knowledge should have, and what functions you can call on it. (This is a short version of Viliam's reply, I think.)
To elaborate, it's different to say "sometimes you should do X" and "this is the ideal". Like, sometimes I do proofs by contradiction, but not every proof is a proof by contradiction, and so it's just a methodology; but the idea of 'doing proofs' is foundational to mathematics / could be seen as one definition of 'what mathematical knowledge is.'
The example in Beautiful Probability seems underdefined. It doesn't specify whether whether or not the second had the option of stopping before N=100.
I don't know how many people, if any, are actually going around in daily life trying to assign or calculate probabilities (conditional or otherwise) or directly apply Bayes' theorem. However, there are core insights that come from learning to think about probability theory coherently that are extremely non-obvious to almost everyone, and require deliberate practice. This includes seemingly simple things like "Mathematical theorems hold whether or not you understand them," "Questions of truth and probability have right answers, and if you get the wrong answers you'll fail to make optimal decisions," or "It's valuable, psychologically and for interpersonal communication, to be able to assign numerical estimates of your confidence in various beliefs or hypotheses." Other more subtle ones like "it is fundamentally impossible to be 100% certain of anything" are also important, and *much* harder to explain to people who aren't aware of the math that defines the relevant terms.
My day job as a research analyst involves making a lot of estimates about a lot of things based on fairly loose and imprecise evidence. In recent years I've been involved in helping train a lot of my coworkers. I find myself paraphrasing ideas from the Sequences constantly (recommending people read them has been less helpful; most won't, and in any case transfer of learning is hard). I notice that their writing, speaking, and thinking become a lot more precise, with fewer mistakes and impossibilities, when I ask them to try doing simple mental exercises like "In your head, assign a probability estimate to everything you claim will happen or think is true now, and add appropriate "likeliness" quantifiers to your sentences based on that."
Also, I've had multiple people tell me that they won't, or even literally can't, make numerical assumptions and estimates without numerical data to back them up, sometimes with very strict ideas about what counts as data. The fact they their colleagues manage to make such assumptions and get useful answers isn't enough to persuade them otherwise. Math is often more likely to get through to such people.
(Not speaking for Eliezer, obviously.) "Carefully adjusting one's model of the world based on new observations" seems like the core idea behind Bayesianism in all its incarnations, and I'm not sure if there is much more to it than that. The stronger the evidence, the more signifiant the update, yada-yada. It seems important to rational thinking because we all tend to fall into the trap of either ignoring evidence we don't like or being overly gullible when something sounds impressive. Not that it helps a lot, way too many "rationalists" uncritically accept the local egregores and defend them like a religion. But the allegiance to an ingroup is emotionally stronger than logic, so we sometimes confuse rationality with rationalization. Still, relative to many other ingroups this one is not bad, so maybe Bayesianism does its thing.
From https://en.wikipedia.org/wiki/Egregore#Contemporary_usage
a kind of group mind that is created when people consciously come together for a common purpose
I am new to this stuff but did we not have like 200 years of observations about Newton's theories? How would have a Bayesian adjusted their models here? I use this example as a "we now know better" - Is it the "new" observation that is key?
My impression from the Sequences seems to be that Eliezer considers Bayesianism to be a core element of rationality. Some people have even referred to the community as Bayesian Rationalists. I've always found it curious, like it seemed like more of a technicality most of the time. Why is Bayesianism important or why did Eliezer consider it important?