I'm writing a more in-depth review of the State of Feedbackloop Rationality. (I've written a short review on the original post)

But I feel like a lot of people have some kind of skepticism that isn't really addressed by "well, after 6 months of work spread out over 1.5 years, I've made... some progress, which feels promising enough to me to keep going but is not overwhelming evidence if you're skeptical."

So, this post is sort of a rant about that. 

A lot of people have default skepticism about "the feedbackloop rationality paradigm" because, well, every rationality paradigm so far has come with big dreams and underdelivered. I, too, have big dreams, and I am probably not going to achieve them as much as I want. That's not a crux^[1] for me.

A lot of people also have some notion that "building multiple feedback loops" is a kind of elaborate meta that puts your head in the clouds. Dudes, the whole point of multiple feedback loops is to put you in direct contact with reality in as many ways as practical.

The thing that drives me nuts is, is that the basic premise here is:

If you put in a lot of effortful focused, practice...
...on the edge of your ability
...with careful attention to immediate feedback
...and periodic careful attention to "is this process helping my longterm goals?"

...you will get better at thinking.

(And, if a lot of people systematically do that and we track what works for them, we can get from the domain of Purposeful Practice to Deliberate Practice^[2])  

This seems so reasonable. This really is not promising a free lunch.

(I should clarify: I am not saying that there is robust scientific evidence for deliberate practice working for thinking. There is not – the evidence for meta-learning is generally confused and sucks and there particularly is not evidence that deliberate purposeful practice works on openended conceptual questions. I'm saying "c'mon, what the fuck do you think will work better at improving your ability to think? a) careful, focused practice on the edge of your ability with careful attention to feedback b) just generally trying to do object level work in domains where you ~never get much feedback on whether you're really accomplishing your ultimate goals?")

There are a lot of followup questions that aren't as obvious, but it seems crazy to not believe that. And if you took it seriously, I don't think the right response is to shrug, and wait for Ray to come back with some visibly impressive results. 

I am, in fact, trying to get some visibly impressive object-level shit done so that skeptical people say "okay I guess the evidence is enough to warrant a more serious look." If maintain a lot of momentum, I might be at that point in ~a year. Figuring out how to do that without burning out is kinda hard (which is what I'm currently working on).

But, I feel like I really shouldn't have to do this, to argue the claim: "effortful, focused practice at thinking with careful attention to feedback will pay off" and "improved thinking is important for making plans about x-risk and AI safety."

Some people followup with "I believe Ray found something that worked for Ray, but don't believe it'll generalize." Which is also a sort of fair response to generically finding out some guy is out there inventing a rationality paradigm without knowing any details. But, c'mon guys, the whole point of the Feedbackloop Rationality Paradigm is to improve your own feedbackloops with your own judgment, with a mixture of legible arguments as well as feelings in your heart. 

I'm annoyed about it because, there's a lot of people around who seem pretty smart in many ways, who I think/hope can help humanity thread the needle of fate, but seem at least somewhat metacognitively dumb sometimes in ways that seem fixable. They bounce off for various reasons. (I get this more from established professionals in the x-risk space, than from median LessWrong folk who generally seem excited for some kind of rationality training content.)

If I ask them why, they mention some arguments that seems like, sure, a reasonable thing to think about (i.e. "if you focus on feedback you might Goodhart yourself", or "I don't have hours and hours to spend doing toy exercises."). But they don't spend 5 minutes thinking about how to address those problems, which IMO seem totally possible to address. 

I know a small number of people who seem to have actually constructed some kind of practice regimen for themselves (which might look very different from mine). This rant is not directed at them. (I do think those people often could use more deliberate attention on what sort of practice they do, and how often they do it. But, I'm not mad about it)

Is the evidence for this ironclad? No. But neither is the evidence that just focusing on your object level work is going to be enough to steer towards the most important questions and approaches in time. And... I just roll to disbelieve that your true objection is that you don't believe in purposeful practice.

FAQ on some things that seem reasonable to be skeptical about

I don't think practicing "better thinking" will help me. I think I should focus on domain-specific skills.

For many people – yep, that sounds correct. Literally the first post in the Feedbackloop rationality sequence is Rationality !== Winning, where I note that even if rationality can help you with your problems, if you have pretty ordinary problems, it's not obviously the right tool for the job.

I'm focused on "think that helps you solve confusing problems, for which there is no established playbook."

With AI safety in particular, I think you should be worried that the job is confusing, such that it's not obvious what the right tool for the job is. We've never demonstrably stopped an unfriendly AI FOOM or a slow-rolling loss of power before. I think you should have a lot of uncertainty about what the right approaches are.

We know how to mentor people into specific research paradigms, but AFAICT humanity doesn't have much systematized training on how to figure out which research field (if any) is coming at things from the right angle.

I'm particularly worried that a lot of people are just following a gradient of learning ML, assuming the ML research paradigm is basically the right lens, without cultivating the ability to form good taste on whether that's true. (And, for that matter, that other people are following a gradient of "learn whatever math happened to be involved with existing agent foundations research and try to get better at that.")

But, you can't practice the most important parts of thinking, for preparadigmatic research. Focusing on feedback loops will lead you to goodhart.

I think if you sit and think about it, with an attitude of "how can I solve this problem?", rather than "this seems obviously hopeless", it is really not that hard to z

There’s a cluster of skills necessary for tackling a lot of “real x-risk work”, or really, any kind of pursuing a difficult, confusing goal. Some examples:

• Noticing you are confused
• Noticing you are still confused
• Sitting with the discomfort of being confused and also not sure if your project is on the right track for a long time.
• Generating situation-appropriate-strategies for dealing with that confusion and tractionlessness.
• Cultivating open curiosity
• Finding questions worth asking.
• Noticing that your strategy was subtly wrong, and figuring out a way to make it work.
• Grieving for the fact that your current project was fundamentally flawed and you wasted a lot of time, and moving on
• Noticing you are missing a skill, or knowledge, and figuring out how to quickly gain that skill/knowledge.
• Noticing when your plans are resting on shaky assumptions
• Validating those assumptions early, if you can.
• Developing a calibrated sense of when you’re meandering thought process is going somewhere valuable, vs when you’re off track.

I think all of those are trainable skills. Yes, it's possible to misgeneralize from toy problems to real problems, but I think the general engine of "practice on a variety of Toy Confusing Problems, and develop a sense of the failure modes you run into there, and how to account for them" just makes a pretty solid foundation for tracking the subtler or deeper versions of those problems that you run into in the wild.

Is it possible to overfit on your training? Yes, that's why I recommend a variety of types of confusing challenges, so you can sus out which skills actually generalize. 

Maybe, but this will just take so much time. Why is this going to pay off enough to be better than object level research?

This is the question that feels most reasonable to me. Purposeful Practice requires peak cognitive hours, and you don't have that many of those.

My main answer is: Identifying skills that you need day-to-day for your most important projects, and finding an angle for practicing with focused attention for ~15 minutes a day (applied to your day job). You keep your peak hours focused on your primary goals, with a bit of time directed at "how can I push the frontier of my thinking capacity?"

It seems reasonable to spend those 15 minutes on a mix of domain-specific skills and higher level decisionmaking. I do my personal practice of a mix of "code debugging" (a naturally occurring rationality challenge) and "deciding what to do today."

Some details from my self-review of the feedbackloop rationality post

Fluent enough for your day job. 

The primary aim of my workshops and practice is to get new skills fluent enough that you can appy them to your day job, because that's where it's practical to deliberate practice in a way that pays for itself, rather than being an exhausting extra thing you do.

"Fluency at new skill that seem demonstrably useful" is also a large enough effect size that there's at least something you can measure near term, to get a sense of whether the workshop is working.

Five minute versions of skills. 

Relatedly: many skills have elaborate, comprehensive versions that take ~an hour to get the full value of, but you're realisitically not going to do those most of the time. So it's important to boil them down into something you can do in 5 minutes (or, 30 seconds).

Morning Orient Prompts.

A thing I've find useful for myself, and now think of as one of the primary goal of the workshop to get people to try out, is a "morning orient prompt list" that you do every day.

It's important that it be every day, even when you don't need it too much, so that you still have a habit of metacognition for the times you need it (but, when you don't need it too much, it's fine/good to do a very quick version of it)

It's useful to have a list of explicit prompts, because that gives you an artifact that's easier to iterate on.

Also, consider at least one extended deliberate practice push

I think ~20 hours is often enough to get "n00b gains" in a new specific skill. My personal experience is that ~40 hours has been enough to take a place where I had plateau'd, and push through to a demonstrably higher level.

I think it's worth doing this at least once and maybe twice, in a continuous marathon focus, to get a visceral sense of how much work is involved, and which sorts of things actually help you-in-particular. But, this is a lot more expensive, and I don't begrudge people who say "well, that's more than I can commit."

Realistically, I know I won't stick to this sort of thing, and even if it'd work if I stick to it, in practice it'll just sort of sputter and fail. Which doesn't seem worth it.

Sigh, nod. This is indeed the reason that deliberate practice kinda sucks which is why it's underinvested in which is why I think there's alpha in it. 

Generally my answer to this is to a) try to build structures that generally help people with this, b) also be investing in a Thinking Assistant ecosystem that can help people with general focus, of which "help do your serious practice."

I have limited time so I don't have generally open offers to help with this, but my current process is "after you've come to a workshop, you're invited to Deliberate Practice club which meets ~once a month, and charges money to keep it sustainable for me + filter for seriousness", where we both orient on "How did the last month ago? Are we on track 

Okay but I bet Ray is doing some oddly specific stuff that works for Ray and don't really think it'll help me.

Maybe! I think people's cognitive styles do vary.

I feel pretty happy if people read this, and go off on their own to construct their own practice regimens that work for them.

I do think I've got a bunch of domain expertise by now in "how to construct exercises on-the-fly that help people train skills that are relevant to them." At my workshops, I  explicitly tell people "you can do a different exercise, if a given session doesn't feel useful to you." A problem is that designing exercises for yourself is itself a skill, and most people end up doing something not-too-productive the first couple times they try. But I generally try to be building containers that help people put in the practice, wherever they're currently at.

I do have a specific vibe though. I'm working on being more flexible about it. (Someone said "I kinda wish there was a Logan-y version of this workshop" and I think that's a pretty legit thing to want in the world)

The skill ceiling here doesn't seem that high, I'm already pretty competent, it doesn't seem worth it.

Maybe. I don't know you. Maybe you've consumed all the low-and-medium hanging fruit here.

But, idk. 10x programmers seem to exist. 10x UI designers seem to exist. When I look at the gears of how people seem to struggle with it, it seems to me pretty likely that 10x confusing-problem-navigation should be possible. There are tons of subtly wrong ways to make plans, and tons of subtly-or-not-so-subtly dumb ways of sticking with them too long.

I realize that is not that good an argument if you don't already share my intuitions here. But I am interested in hearing from specific people about what-they-think-they-know-and-why-they-think-they-know it about how much room for improvement they think they have.

Why (exactly) won't it work (for you, specifically)

One of my favorite frames/prompts I've found in the past year is "Very specifically, why is this impossible?". And then, for each of those reasons

I've seen a few people respond to this with some kind of broad dismissal.

I am interested in hearing critiques from people who've set, like, at least a 15 minute timer to sit and ask themselves, "Okay, suppose I did want to improve at these sorts of skills, or related ones that feel more relevant to me, in a way I believed in. What concretely is hard about that? Where do I expect it to go wrong?", and then come back with something more specific than "idk it just seems like this sort of thing won't work."

And then, for each of those reasons, ask "okay, why is that impossible?"

And sometimes you're left with pieces that are actually impossible. Sometimes you are left with pieces that are merely "very hard." And sometimes, as soon as you sit and think about it for 2 seconds, you go "oh, well, okay obviously I could do that, it'd just be slightly inconvenient."

I would like to hear critiques from people who have spent at least some time inhabiting the "okay, if I tried to roll up my sleeves and just do this, either for myself, or for contributing somehow to an ecosystem of confusing-problem-solving-training, what actually goes wrong?"

1. ^{^}

   I am holding the big dreams as my polaris because it's helpful for cutting as quickly as possible to whatever the best outcomes turn out to be. "Shoot for the moon and... you'll end up in low-earth orbit, which is pretty cool actually!".

   "Shoot for olympic-level training for x-risk researchers and you'll end up with pretty solid training that gets a bunch of people more traction on the hardest parts of the problem(s)."

2. ^{^}

   Contrasted with Naive Practice, Purposeful Practice involves sustained focus, on skills at the edge of your ability, with explicit goals, paying careful attention to feedback.

   Deliberate practice is "Purposeful Practice that we know works." (I claim that the current state of my paradigm is "minimum-viable-deliberate-practice", in that it's been through at least a couple rounds of weeding out stuff that didn't work, and the stuff that remains has worked at least decently)