Review
Thank you for this post!
After reading this, it seems blindingly obvious: why should you wait for one of your plans to fail before trying another one of them?
This past summer, I was running a study on study on humans that I had to finish before the end of the summer. I had in mind two methods for finding participants; one would be better and more impressive and also much less likely to work, while the other would be easier but less impressive.
For a few weeks, I tried really hard to get the first method to work. I sent over 30 emails and used personal connections to try to collect data. But it didn't work. So I did the thing that I thought to be "rational" at the time. I gave up and I sent my website out to some people who I thought would be very likely to do it. Sure enough, they did.
At the time, I thought I was being super-duper rational for allowing my first method to fail (not deluding myself that it would work and thus not collecting any data) and then quickly switching to the other method.
However, after reading this post, I realize that I still made a big mistake! I should have sent it out to as many people as possible all at once. This would have been a bit more work since I would have to deal with more people and they would use a slightly different structure, but I was not time constrained. I was subject constrained.
I'm going to instill this pattern in my mind and will use it when I do something that I think has a decent chance of failing (as my first method did).
You're welcome, and thank you for the example! As you point out, whether you're constrained on time often matters for how many plans you can attempt.
I do hope you give yourself some points for noticing the first method wasn't working and switching. That's better than winding up with no data. I'd encourage thinking of this as an opportunity to get even more points.
How do you manage to avoid Goodharting or quota-filling on other plans?
I ask because this runs to the very core of my current confusion in life: how to brainstorm (or babble) better plans in the first place. It is exceedingly easy for me to abandon one of my plans simply because I had no confidence it was that "good" in the first place. Also, I find it exceedingly easy to generate a dozen different plans if the objective is "just invent other plans" rather than "invent equally good plans". But, if the objective is "invent equally good plans" I groan and cringe and, I'm ashamed to admit, give up.
For example, in the past I was told that if I wanted to be a professional film director then I needed to "enter films into film festivals" I did that to zero success. None. No laurels. Nothing. I had no faith in that plan to begin with and only did it because I was "supposed to do it".
"Putting your films 'out there' (i.e. online)" was the only other plan I had.
It would have been nice to have been able to essay other plans that weren't just quota-fillers.
As for parallel essaying of other plans, it reminds me of an observation David Snowden made in his Cynefin Sensmaking framework, that while Complicated and Simple systems which are highly predictable and therefore usually have a "best practice" or one and most optimal way to do something. Complex systems in which you can only predict trajectories are best suited to launching several simultaneous or parallel interventions which are not "fail safe" but "safe to fail". He means you can wind down or stop the interventions which are less successful as you generate feedback, and scale up on successful ones. Sort of like A/B testing but more like A/B/C/D/E/ testing.
The following works for me. I do not know how well it works for other people.
I separate coming up with ideas and filtering for good ideas. Many of the ideas I come up with are bad ideas. For example, I'm about to spend a few minutes by the clock coming up with plans for becoming a professional film director.
"Go to film school and ask the professors for professional contacts. Show up at a college office hour and as the professors for contacts even if I'm not a student. Make movies in my backyard with home equipment, and put them online. Reach out to film studios and ask if they have educational opportunities or internships. Hire a professional film director to tutor me. Read books and biographies about professional film directors and copy what they did. Hang around the next actor's strike and ask if anyone wants to goof off and do an amateur project that wouldn't break the strike. Make movies and then talk to local theatres and see if they want to run those movies, maybe during slow times/days. Create an online buzz around an obscure and rarely seen auteur film that The Mainstream Media doesn't want you to see. Project it onto the sky on a screen held up by drones. Make a film of a famous fanfic- hey, podfics hitch a ride on someone else's story why not a film."
Okay, time over. Some of those ideas need more detail- like, making a movie with the tools I have comes up as a first step of several plans and I'd need to work through more about how to do that. That's not necessarily a bad thing; I know just enough about film-making to know there's some different choices I'd make while filming depending on the eventual format but maybe I can save some effort with multiple cuts or something. Other ideas are ways to fish for more ideas, like reading books on the subject or asking professors what to do.
Some are just bad ideas. Getting professional film directors to tutor me sounds expensive, and also I'm not at all sure the constraint is lack of skill at directing. Actor's strikes seem to happen about every ten years so maybe keep that in your back pocket while you work on other plans. As for projecting it onto the sky, I don't have any idea how that gets you paid enough to get called a professional plus you might get some kind of public disturbance complaint? But. . . talking to local theatres doesn't seem like a terrible idea? Putting some kind of weird spin on a movie released online might work- it at least points at different tweaks or implementations to just putting it up on Youtube.
The point is, I didn't come up with those plans to fill a quota. I was just coming up with a bunch of ideas. If in the end there was only one good idea in the mix, then I'd use that one and ignore the others. Sometimes there's no good ideas, in which case I give my brain a mental cookie for coming up with ideas and another for not pretending that any of them were good.
I have said in the past that the epiphanies I remember from Harry Potter and the Methods of Rationality aren't the pieces where a long speech was made on a particular point, but in places where a seemingly offhand line provided a bolt of lightning.
"Are you confident in the success of this plan? No, that is the wrong question, we are not limited to a single plan. Are you certain that this plan will be enough, that we need essay no others? Asked in such fashion, the question answers itself. The path leading to disaster must be averted along every possible point of intervention."
I don't remember if my jaw actually dropped on this line, but I do remember pausing, rereading, and then going "oh duh." Somehow I had this idea that when you wanted something to happen you should make a plan, and that plan might have contingencies or fallbacks, and it might fail in which case you'd try something else, but the idea that you could just keep making multiple plans and try more than one at a time hadn't quite occurred to me.
The sazen for this post is "Essay Other Plans." The word "essay" has a second, less common meaning of "try" or "attempt." The point is that you can keep trying things in parallel, especially when those plans don't interfere with each other.
What does this look like?
Usually when I want to do something big, I write down what I'm trying to do at the top. Then I make a bullet pointed list of the steps I think I need to take. I read the steps, and do a little murphyjutsu to figure out where the obvious failure points are or what might go wrong. Places where I know I need to fill in the details later get a pile of question marks surrounding what I do know, and sub bullet points with how I'll find out. There's a separate skill of noticing when I'm not using enough detail, like if my plan involves the step "and then I'll write the login page" or "and then I'll study for the exam" when I need to actually step through how I'm going to do that.
Then I hit enter until I've got a bit of space or just hit whatever key combination creates a new page in my word processor, and I start a new sequence of steps that has nothing in common with the first one except the output. It would be very strange after all if there was exactly one way to achieve a goal. Maybe I want to learn guitar and the first plan was to watch lots of Youtube videos and follow along, and then the second plan is to find a local teacher. Maybe I want a particular piece of software to exist and the first plan is to write it myself, and the second plan is to offer to pay a freelancer. There's no need to stop at two plans by the way — if I've got time sometimes I go full on As Many Ideas on the problem and cook up a couple dozen. The mental motion is a bit like maintaining two hypotheses, except they aren't exclusive.
Speaking of which, that's the next step. I pull up the different plans side by side and see if there are points where they're in obvious conflict. Often my personal time is the limiting factor, since I can only be in one place doing one thing at a given instant. Sometimes I can work around that by enlisting collaborators, but other times that won't work very well. Collaborators themselves are scarce resources.
I'm also usually looking for places where the different plans can reinforce each other. When I was plotting the 2023 LW Community Census, I knew I wanted more responses. I could get that by having more people reading my LessWrong posts, and I could also get it by announcing it on other platforms, and I could also get it by having more lead time and launching the census at a more typical December than in the unusual February. Thing is, writing more LessWrong posts meant more people recognized my username and so might have been happier to help me spread the word, and by launching the census in December I was able to meet some people at Solstice afterparties and ask them to help.
The most useful cases are when plans have a long rest time when you can't meaningfully speed them up. Consider the goal of having a nice, well-baked loaf of bread to bring to a dinner party. Double check your measurements, sure, but also consider making several loaves simultaneously and taking the best looking one. Do both! If you're considering several venues for a meetup event and you'll need to talk to the sites, send them all emails as you find them and let them respond at their own paces.
The thing you don't want to do is stop on the first passable plan that occurs to you.
Sometimes people act like there are 1. bad plans, 2. good plans, and 3. great plans. Then when they try to come up with plans, they keep thinking them up one at a time until they get the first plan that's at least good and then they stop.
(I say "people" there like that wasn't how I generally operated until I had the bolt of lightning from the HPMOR line.)
Imagine sitting down with some dice trying to roll a high number and rolling a 2, a 1, then a 4 and stopping. Sometimes you're on a tight time limit and you only had time for three rolls, sure, but maybe you're not limited. If you're allowed to keep rolling until you decide you're done, why would you stop before you got the highest number possible?
You don't want to forget your plans, and having more than one involves a little more operational work. I'm a tremendous fan of the ancient and powerful technique of writing things down. That ways I don't forget plans as fast as I come up with them. Once I have a list of plans, I can go through them and circle the ones I think are actually good, with extra circles indicating great ideas.
Also, while it's not the main point of this post pay attention for when the same plan can be attempted multiple times. If you're trying to find a job, then "Look up a company you want to work for, then look through your LinkedIn and try to find a connection you have at that company, then ask them for a recommendation" is a plan you can try repeatedly without waiting for one specific attempt to fail. "Plant seeds in the ground, water them, and wait for them to grow" is a good plan and you can plant lots of seeds at once.
Don't be afraid to adjust each plan as you go along, or even abandon some.
Let's say you want to get people interested in some philosophy you think is important. You can write a textbook about it. You can run meetups with free snacks and talk about it in person. You can found a nonprofit to work on new and better forms of the philosophy. You can also write alternate universe fanfiction about characters who believe in the philosophy. You can probably do all of those in parallel!
If some of those plans start working better than others — if not many people show up to your meetups or work on the textbook is slow, but your fanfic is flowing easily and you have thousands upon thousands of readers — then you can pay more attention to the plans that are working. That's a thing you're allowed to do. Just because you made a plan and started working on it doesn't mean the plan needs to be carried out and finished.
Yes, sometimes it makes sense to wind down plans gracefully. Loudly give up, don't quietly fade. But if you're never allowed to give up on a plan, that puts too high a bar on the plans you should try. It's fine to say that an idea isn't working out, or that it's working but it's not the best use of your efforts now.
In most circumstances, you can also move resources or pieces of progress from one plan to another. Imagine you're trying to make friends in a new city. You've tried going to the local rationalist meetup, you've tried hanging out in sports bars, you've tried using dating apps, and you've tried offering to run D&D games. Maybe you made a lot of friends from the rationalist meetups and a couple of friends from the bars, but nobody has shown up to the D&D games or responded on the apps. You can invite the friends from the meetups to the bars or the bars to the meetups if you want.
(I don't just want to make friends so that I have people to play D&D with me, but many people who've met me will confirm that's not entirely wrong either.)
In a way, this is being persistent really fast.
"If at first you don't succeed, try try again" is an aphorism. But why wait until you don't succeed? Maybe that phrase should be "If it's not obvious you'll succeed, try try in parallel."
It has many of the benefits of persistence. You can gain experience with a problem quickly if you're attacking it from so many different angles. You get a lot of exposure with what works better or worse. One advantage is that it takes much less time.
The bigger advantage might be just in how I tend to think and operate. I don't seem to have the same limited mental resources other people describe as willpower or spoons, or at least those are very rarely the limitation. Instead I seem to have mental momentum. When I'm interested in something and working on it, I want to work on it more and more and more. The worst thing I can do to my ability to get things done is to stop, which means even if I can't meaningfully make any more progress at the moment putting it down is kind of like applying the brakes on a bicycle. It's easier to maintain speed or accelerate a little than it is to start moving from a standing stop.
If I put down one plan and pick up another plan for the same goal though? That doesn't hit the brakes. Projects where I can hop from plan to plan let me stay in the same mindspace, losing less speed. I don't know how typical my mind is here, but I appreciate not feeling trapped based on the success or failure of one plan. It's less stressful to contemplate one of the plans failing, like I have a line of retreat.
"It is possible that you have already done everything you can. Yet I find this a very rare event indeed, and more often said than done. I suspect rather that you have only done what you customarily do."
-Harry Potter and the Methods of Rationality, Chapter 92
It often makes sense to try the obvious, safe paths first. But if you only think to try other paths after the obvious paths have failed, then you lost a lot of time. If nothing else, you can think about what you'll do if the obvious path fails now, before you know if it worked or not, and if any of those ideas seem good then you can try them now before you have to wait and find out if the obvious path is going to fail. I'm not saying don't do the obvious plans! Obvious plans are usually obvious for a good reason! But you're quite possibly not limited to one plan at a time!
Essay other plans.