I think "Rationality is winning" is a bit of a trap. 

(The original phrase is notably "rationality is systematized winning", which is better, but it tends to slide into the abbreviated form, and both forms aren't that great IMO)

It was coined to counteract one set of failure modes - there were people who were straw vulcans, who thought rituals-of-logic were important without noticing when they were getting in the way of their real goals. And, also, there outside critics who'd complain about straw-vulcan-ish actions, and treat that as a knockdown argument against "rationality."

"Rationalist should win" is a countermeme that tells both groups of people "Straw vulcanism is not The Way. If you find yourself overthinking things in counterproductive ways, you are not doing rationality, even if it seems elegant or 'reasonable' in some sense."

It's true that rationalists should win. But I think it's not correspondingly true that "rationality" is the study of winning, full stop. There are lots of ways to win. Sometimes the way you win is by copying what your neighbors are doing, and working hard. There is rationality involved in sifting through the various practices people suggest to you, and figuring out which ones work best. But, the specific skill of "sifting out the good from the bad" isn't always the best approach. It might take years to become good at it, and it's not obvious that those years of getting good at it will pay off.

Rationality is the study (and applied skill) of finding cognitive algorithms that form better beliefs and make better decisions. Sometimes this is the appropriate tool for the job, and sometimes it's not. 

The reason this is particularly important is in deciding what feedback loops to focus on when developing rationality. If you set your feedback loops as "am I winning, generally?" (or, "are the students of my rationality curriculum winning, generally?"), well, that's a really noisy feedback loop that's probably swamped by random environmental variables. It's not nothing, but it's really slow.

There's also a problem where, well, a lot of techniques that help with winning-at-life just aren't especially about rationality in particular. If you're running a rationality program with "win-at-life" as your goal, you may find yourself veering in a direction that's not really capitalizing on the things rationality was actually especially good at, and become a generic self-help program. Maybe that's fine, but the result seems to lose something of the original spirit.

The domains where rationality matters are domain where information is scarce, and the common wisdom of people around you is inadequate. 

The coronavirus pandemic was a good example where rationality was relevant: a very major change disrupted society, there was not yet a scientific consensus on the subject, there were reasons to doubt some claims by scientific authorities, and your neighbors were probably slow to react. (I think rationalists did well at navigating the early pandemic, but, alas, also stayed in overly-stressful-lockdown-mode longer than was appropriate, and lacked some other key skills)

Building a startup is a domain where I think rationality is pretty relevant. There is a lot of common wisdom that is relevant. Paul Graham et al have useful advice. But because you need to outperform a lot of competitors, common wisdom isn't enough. You need to continuously model the world, design products people don't know they want yet while soaking in new information so you can continuously update and iterate. You need to stare into the darkness and admit major mistakes.

Rationality is particularly helpful for solving problems we don't know how to solve.

Rationality is useful for judges and jurors, who must accurately weigh evidence and not get caught up in misleading arguments. (I know a lawyer who tried to explain Bayes theorem to a court, and the judge/jury... didn't believe him)

Rationality and AI alignment are heavily intertwined on LessWrong, and a major reason for that is catastrophic misalignment is a problem we won't get to see and iterate on with the usual set of scientific wisdom.

Eliezer notes a few mistakes he made in the original sequences:

It was a mistake that I didn’t write my two years of blog posts with the intention of helping people do better in their everyday lives. I wrote it with the intention of helping people solve big, difficult, important problems, and I chose impressive-sounding, abstract problems as my examples.

In retrospect, this was the second-largest mistake in my approach. It ties in to the first-largest mistake in my writing, which was that I didn’t realize that the big problem in learning this valuable way of thinking was figuring out how to practice it, not knowing the theory. I didn’t realize that part was the priority; and regarding this I can only say “Oops” and “Duh.”

Yes, sometimes those big issues really are big and really are important; but that doesn’t change the basic truth that to master skills you need to practice them and it’s harder to practice on things that are further away. (Today the Center for Applied Rationality is working on repairing this huge mistake of mine in a more systematic fashion.)

A third huge mistake I made was to focus too much on rational belief, too little on rational action.

I agree with these mistakes, and I think it was probably right for some amount of correction towards rationality-solving-your-personal-life-problems. Rationality does help with personal life problems. It helps you understand and get unconfused about your goals, understand the perspectives of people you need to interact with, etc. I have definitely gotten value out of CFAR-style workshops. I'd certainly rather have overcorrected in this direction than not at all. 

(I also want to be clear, I don't think CFAR was just focused on self-help-y stuff. A lot of it continues to be pretty foundational to my strategic thinking in a variety of domains. I think double-crux ended up being pretty important to navigating disagreement at Lightcone)

But, 

Well, 

It's hard to say this kindly. I think "rationality-for-solving-personal-problems" created a cultural vortex that attracted people with personal problems. And I think this diverted attention away from "use rationality to systematically be faster-than-science at understanding and solving difficult problems." 

I'm glad those people (including me, to some degree) found/built a home. But I want to recapture some of that original spirit.

I had a bit of a wake-up-call a few years ago when I read How to Measure Anything (see Luke's review). This is not a book written to create a rewarding community of nerds. It's targeted at professional businessmen who want to improve decisions where thousands (or millions) of dollars are at stake. It teaches Bayes and calibration training and value-of-information in a context that does imply concrete exercises, but, is not necessarily for everyone. It presents rationality as a professional tool that is useful for people that want to specialize in some types of problems.

How to Measure Anything notably won't help you (much) with aligning AI. It teaches "good decisionmaking", but doesn't teach "research taste in novel domains". I think there's a concrete branch of rationality training that'd be relevant for novel research, that requires pretty different feedbackloops from the "generally be successful at life" style of training. I think some of "research taste rationality" is reasonably alive in academia, but many elements are not, or are not emphasized enough.

I'm interested in cultivating a set of rationality feedbackloops that are geared towards training research taste. And as I've mulled this over, I've found it helpful to distance myself a bit from "rationality is systematized winning" as a catchphrase.

Appendix: Being Clearer on My Claims

Originally I stopped the post here. But commenter df fd asked:

I got into Rationality for a purpose. If it is not the best way to get me to that purpose [i.e. not winning] then Rationality should be casted down and the alternative embraced. 

And this made me realize I had left my claims here somewhat vague, and a lot of my background models implicit. So here's a breakdown of what I mean and why I care:

...

First, I want to note my definition of rationality here is not new, it's basically how it was described by Eliezer in 2012, and I'm pretty confident it's what he meant when writing most of the sequences. "Eliezer said so" isn't a great argument, but it looks like some people might feel like I'm shifting goalposts and changing definitions and I claim I am not doing that. From Rationality: Appreciating Cognitive Algorithms:

The word 'rational' is properly used to talk about cognitive algorithms which systematically promote map-territory correspondences or goal achievement. 

He notes that in the sentence "It's rational to believe the sky is blue", the word rational isn't really doing any useful work. The sky is blue. But by contrast, the sentence "It's (epistemically) rational to believe more in hypotheses that make successful experimental predictions" is saying something specifically about how to generate beliefs, and if you removed "rational" you'd have to fill in more words to replace it.

I think there's something similar going on with "it's rational to [do an object-level-strategy that wins]." If it just wins, just call it winning, not rationality. 

I do agree Eliezer also described rationality as systemized winning, and that ultimately winning was an important arbitrator of "were you being rational", and that this required breaking out of some narrower definitions of rationality. I think these are not in as much tension as they look, but they are in at least superficial tension. 

With that as background, here's the claims I'm actually making:

...

Second: I'm mostly making an empirical claim as to what seems to happen to individual people (and more noticeably to groups-of-people) if they focus on the slogan "rationality is winning." 

It's hypothetically possible for it to true that "rationality is systemized winning", and for it to be subtly warping to focus on that fact. The specific failure modes I'm worried about are:

• The feedbackloops are long/slow/noisy, which makes it hard to learn if what you're trying is working.
• If you set out to systematically win, many people end up pursuing a lot of strategies that are pretty random. And maybe they're good strategies! But bucketing all of them under "rationality" starts to deflate the meaning of the word.
• People repeatedly ask "but, isn't it rationality to believe false things?". And, my answer is "maybe, for some people? I think you should be really wary of doing that, but there's certainly no law of the universe saying it's wrong." But, this is particularly bad as a way to orient as a group. The first generation of people who came for the epistemics maybe has a decent judgment on when it's okay to ignore epistemics. The second generation who comes for "systemized winning, including maybe ignoring epistemics?" has less ability to figure out if they're actually winning because they can't reason as clearly.
• Similarly and more specifically: a lot of things-that-win in some respects are wooy, and while I think there's in fact good stuff in some woo, while the first generation of rationalists exploring that woo were rationalists with a solid epistemic foundation. Subsequent generations came more for the woo than for the rationality (See Salvage Epistemology).
• In both the previous two bullets, the slogan "rationality is winning" is really fuzzy and makes it harder to discern "okay which stuff here is relevant?". Whereas "rationality is the study of cognitive algorithms that systematically arrive at truth and succeeding at your goals" at least somewhat 

...

Third: The valley of bad rationality means that study of systemized winning is not guaranteed to actually lead to winning, even en-net over the course of your entire lifetime. 

Maybe civilization, or your local culture, just has too many missing pieces for the deliberate study of systematic winning to be net-positive. Or maybe you can make some initial progress, but hit a local optima, and the only way to further improve is to invest in skills that will take too long to pay off.

...

Fourth: Honestly, while I think LessWrong culture is good at epistemics, addressing motivated cognition, and some similar things... I don't have a strong reason to believe that we are particularly good at systematically winning across domains (except in domains where epistemics are particularly relevant)

So while it might be true that "The True Spirit of Rationality" is systemized winning, and epistemics is merely subservient to that... it's nonetheless true that if you're showing on LessWrong or in other rationalist spaces, I think you'll be kind of disappointed if you're hoping to learn skills that will be help you win at life in a generalized sense. 

I do still think "more is possible". And I think there is "alpha" in epistemics, such that if you invest a lot in epistemics you will find a set of tools that the rest of the world is less likely to find. But I don't have a strong belief that this'll pay off substantially for any specific person.

(side note: I think we have maybe specialized reasonably in "help autistic nerds recover their weak spots", which means learning from our practices will help with some initial growth spurt, but then level up)

...

So, fifth: So, to answer df fd's challenge here: 

I got into Rationality for a purpose if it is not the best way to get me to that purpose [i.e. not winning] then Rationality should be casted down and the alternative embraced. 

A lot of my answer here is "sure, that might be fine!" I highly recommend you focus on winning, and use whatever tools are appropriate, which sometimes will be "study/practice cognitive algorithms" shaped and sometimes will have other shapes. 

I do agree there is a meta-level skill of figuring out what tools to use, and I do think that meta-level skill is still pretty central to what I call rationality (which includes "applying cognitive algorithms to make good decisions"). Having that skill is good, all-else being equal. But it's not necessarily the case that studying that skill will pay off.

Linguistically, I think it's correct to say "the rational move is the one that resulted in you winning (given your starting resources, including knowledge)", but, "that was the rational move" doesn't necessarily equal "'rationality' as a practice was helpful."

Hope that helps explain where I'm coming from.