Earlier this year I spent a lot of time trying to understand how to do research better. This post was one of the few resources that actually helped. It described several models that I resonated with, but which I had not read anywhere else. It essentially described a lot of the things I was already doing, and this gave me more confidence in deciding to continue doing full time AI alignment research. (It also helps that Karnofsky is an accomplished researcher, and so his advice has more weight!)
I've claimed that the best way to learn is by writing about important topics. (Examples I've worked on include: which charity to donate to, whether life has gotten better over time, whether civilization is declining, whether AI could make this the most important century of all time for humanity.)
But I've also said this can be "hard, taxing, exhausting and a bit of a mental health gauntlet," because:
This post goes through some tips I've collected over the years for dealing with these sorts of challenges - both working on them myself, and working with teammates and seeing what works for them.
A lot of what matters for doing this sort of work is coming at it with open-mindedness, self-criticality, attention to detail, and other virtues. But a running theme of this work is that it can be deadly to approach with too much virtue: holding oneself to self-imposed deadlines, trying for too much rigor on every subtopic, and otherwise trying to do "Do everything right, as planned and on time" can drive a person nuts. So this post is focused on a less obvious aspect of what helps with wicked problems, which is useful vices - antidotes to the kind of thoroughness and conscientiousness that lead to unreachable standards, and make wicked problems impossible.
I've organized my tips under the following vices, borrowing from Larry Wall and extending his framework a bit:
Overall, it's tempting to try to "boil the ocean" and thoroughly examine every aspect of a topic of interest. But the world is too big, and the amount of information is too much. I think the only way to form a view on an important topic is to do a whole lot of simplifying, approximating and skipping steps - aiming for a step of progress rather than a confident resolution.
Laziness
Hypothesis rearticulation
My previous piece focused on "hypothesis rearticulation": instead of defending what I was originally going to argue, I just change what I'm arguing so it's easier to defend. For example, when asking Has Life Gotten Better?, I could've knocked myself out trying to pin down exactly how quality of life changed in each different part of the world between, say, the year 0 and the year 1000. Instead, I focused on saying that that time period is a "mystery" and focused on arguing for why we shouldn't be confident in any of a few tempting narratives.
My previous piece has another example of this move. It's one of the most important moves for answering big questions.
Questions for further investigation
This is really one of my favorites. Every GiveWell report used to have a big section at the bottom called "Questions for further investigation." We'd be working on some question like "What about the possibility that paying for these services (e.g., bednets) just causes the government to invest less in them?" and I'd be like "Would you rather spend another 100 hours on this question, or write down a few sentences about what our best guess is right now, add it to the Questions for Further Investigation section and move on?"
To be clear, sometimes the answer was the former, and I think we eventually did get to ~all of those questions (over the course of years). But still - it's remarkable how often this simple move can save one's project, and create another fun project for someone else to work on!
What standard are we trying to reach? How about the easiest one that would still be worth reaching?
If you're writing an academic paper, you probably have a sense of what counts as "enough evidence" or "enough argumentation" that you've met the standards for a successful paper.
But here I'm trying to answer some broad question like "Where should I donate?" or "Is civilization declining?" that doesn't fit into an established field - and for such a broad question, I'm going to run into a huge number of sub-questions (each of which could be the subject of many papers of its own). It's tempting to try for some standard like "Every claim I make is supported by a recognizably rigorous, conclusive analysis," but that way madness lies.
I think it's often better to aim for the minimum level of rigor that would still make something "the best available answer to the question." But I'm not absolutist about that either - a frustrating aspect of working with me on problems like this is that I'll frequently say things like "Well, we don't need to thoroughly answer objection X, but we should do it if it's pretty quick to do so - that just seems like a good deal." I think this is a fine way to approach things, but it leads to shifting standards.
Here's a slightly more nuanced way to think about how "rigorous" a piece is, when there's no clear standard to meet. I tend to ask: “How hard would it be for a critic to demonstrate that the writeup's conclusion is significantly off in a particular direction, and/or far less robust than the writer has claimed?”1
The "how hard" question can be answered via something like:
I seem to recall that with GiveWell, we got a lot more successful (in terms of e.g. donor retention) once we got to the point where we could get through an hour-long Q&A with donors with a “satisfying” response to each question - a response that demonstrated that (a) we had thought about the question more/harder than the interlocutor; (b) we had good reason to think it would take us significant time (say 10-100 hours or more) to get a better answer to the question than we had. At this point, I think the C-hardness was at least 10 hours or so - no small achievement, since lots of not-very-informed people know something about some random angle.
(By now, I'd guess that GiveWell’s A-hardness is over 100 hours. But a C-hardness of 10 hours was the first thing to aim for.)
These standards are very different from something like “Each claim is proven with X amount of confidence.” I think that’s appropriate, when you keep in mind that the goal is “most thoughtful take available on a key action-guiding question.”
Impatience
Many people dream of working on a project that puts them in a flow state:
But if you're working on wicked problems, I recommend that you avoid flow states, nice though they may be. (Thanks to Ajeya Cotra for this point.) Maybe you instead want a Harrison Bergeron state: every time you're getting in a groove, you get jolted out of it, completely lose track of what you were doing, and have to reassemble your thoughts.
That's because one of the most productive things you can do when working on a wicked problem is rethink what you're trying to do. The more you interrupt yourself, and the less attached you are to the plan you had, the more times you'll be able to notice that what you're writing isn't coming out as planned, and you should change course.
Checkins and virtual checkins
I think the ideal way to interrupt yourself is to be working with someone else who's engaged in your topic and has experience with similar sorts of work (at Open Philanthropy, this might mean your manager), and constantly ping them to say things like:
I think this is helpful for a few reasons:
With early GiveWell, I got a huge amount of value from Elie, who consistently wanted to do things far less thoroughly than I wanted to. I probably ended up doing things 3x as thoroughly as he wanted and 1/3 as thoroughly (and so 3x faster!) as I originally wanted - a nice compromise.
These kinds of checkins can be very vulnerable (especially when the topic is something like "I can't accomplish what we both said I would"), and it can be hard to have the kind of relationship that makes them comfortable. It's best if the manager or peer being checked in with starts from a place of being nonjudgmental, remembering the wicked nature of the problem and not being attached to the original goals.
I also recommend imagining an outsider interrupting you to comment on your work - I think this can get you some of those same benefits.
Outline-driven research
I recommend always working off of a complete outline of what you are going to argue and how, which has ideally been reviewed by someone else (or your simulation of someone else) who said "Sure, if you can defend each subpoint in the way you say you can, I'll find this valuable."
That is:
Constantly track your pace
I think it's good to consistently revisit your estimate of how quickly you're on pace to finish the project. Not how quickly you want to finish it or originally said you would finish it - how quickly it will be finished if you do all of the remaining sections at about the pace you've done the current ones.
I think a common issue is that someone looks very thoroughly into the first 2-3 subquestions that come up, without noticing that applying this thoroughness to all subquestions would put the project on pace to take years (or maybe decades?) Consistently interrupting yourself to re-estimate time remaining can be a good prompt to re-scope the project.
Don't just leave a fix for later; duct tape it now
This tip comes from Ajeya. When you reach some difficult part of the argument that you haven't thought about enough yet, it's tempting to write "[to do]" and figure you'll come back to it. But this is dangerous:
Instead, write down the shortest, simplest version of the point you can - focusing on what you currently believe rather than doing a fresh investigation. When you read the piece over again later, if you're not noticing the need for more, then you don't need to do more.
Hubris
Your take is valuable
A common experience with this kind of work is the "too-weak wrong turn": you realize just how much uncertainty there is in the question you're looking into, and how little you really know about it, and how easy it would be for someone to read your end product and say things like: "So? I already knew all of this" and "There's nothing really new here" and "This isn't a definitive take, it's a bunch of guesswork on a bunch of different topics that an expert would know infinitely more about" and such.
This can be demoralizing to the point where it's hard to continue writing, especially once you've put in a lot of time and have figured out most of what you want to say, but are realizing that "what you want to say" is covered in uncertainty and caveats.
It can sometimes be tempting to try to salvage the situation by furiously doing research to produce something more thorough and impressive.
When someone (including myself) is in this type of situation, I often find myself saying the following sort of thing to them:
I think it can be useful to just take "My take on this topic is valuable" as an almost axiomatic backdrop (once one's take has been developed a bit). It doesn't mean more research isn't valuable, but it can shift the attitude from "Furiously trying to find enough documentation that my take feels rigorous" to "Doing whatever extra investigation is worth the extra time, and otherwise just finishing up."
Your productivity is fine
Understanding deadlines. One of the hardest things about working on wicked problems is that it's very hard to say how long a project is supposed to take. For example, in the first year of GiveWell:
The "deadline miss" didn't come from not working hard, it came from having no idea how much work was ahead of us.
Working on wicked problems means navigating:
Early in a project, I suggest treating a deadline mostly as a "deadline to have a better deadline." Something like: "According to my wildly uninformed guess at how long things should take, I should be done by July 1; hopefully by July 1, I will be able to say something more specific, like 'I've gone through 1/3 of my subquestions, and the remaining 2/3 would take until September 1, which is too long, so I'm re-scoping the project.'"
At the point where one can really reliably say how much time should be remaining, I think one is usually done with the hardest part of the project.
For these sorts of "deadline to have a deadline"s, I tend to make them comically aggressive - for example, “I’m gonna start writing this tomorrow and have it done after like 30 hours of work,” while knowing that I’m actually several months from having my first draft (but that going in with the attitude “I’m basically done already, just writing it down” will speed me up a lot by making me articulate some of the key premises). So I'm both setting absurd goals for what I can accomplish, and preparing to completely let myself off the hook if I fail. Hubris.
Understanding procrastination/incubation. For almost anyone (and certainly for myself), working on wicked problems involves a lot of:
(More at my previous piece.)
In fact, early in the process of working on a wicked problem, I think it's often unrealistic to put in more than a few hours of solid work per day - and unhelpful to compare one's productivity to that of people doing better-defined tasks, where the goals are clear and don't change by the hour.
Working on wicked problems can often be a wild emotional rollercoaster, with lots of moments of self-loathing over being unable to focus, or missing a "deadline," or having to heavily weaken the thing one was trying to say.
It's a tough balance, because I think one really does need to pressure oneself to produce. But especially once one has completed a few projects, I think it's feasible to be simultaneously "failing to make progress" and "knowing that one is still broadly on track, because failing to make progress is part of the process." I think it's sometimes productive to have a certain kind of arrogance, an attitude like: "Yes, I cleared the whole day to work on this project and so far what I have done is played 9 hours of video games. But the last 5 times I did something like this, I was in a broadly similar state, and then got momentum and finished on time. I'm doing great!" The balance to strike is feeling enough urgency to move through the whole procrastinate-produce-rethink process, while having a background sense that "this is all expected and fine" that can prevent excessive personal shame and fear from the "procrastination" and "rethink" parts.
(Personally, I often draft a 15-page document by spending 4 hours failing to write the first paragraph, then 1 hour on the first paragraph, then 1 hour failing to continue, then 1 hour on the rest of the first page, then 4 hours for the remaining 14 pages. If someone tries to interrupt me during the first 4 hours, I tell them I'm working, and that's true as far as I'm concerned!)
Self-preservation
As noted above, working on wicked problems often involves long periods of very low output, with self-imposed deadlines creeping up. This sometimes leads people to try to make up for lost time with a "heroic" effort at superhuman productivity, and to try to handle the hardest parts of a project by just working that much harder.
I'm basically totally against this. An analogy I sometimes use:
A: Fly away as fast as he can, optionally shrieking in terror and letting all the onlookers say “Wow, what a coward.” This is a terrible time to be brave and soldier on! There are so many things Superman can do to be helpful - the single worst thing he can do is go where he won’t succeed.2
If the project is taking "too long," it might be because it was impossible to set a "schedule" for in the first place, and trying to finish it off at a superhuman pace could easily just leave you exhausted, demoralized and still not close to done. Additionally, the next task sometimes seems "scary" because it is actually a bad idea and needs to be rethought.
I generally advise people working on wicked problems to aim for "jogging" rather than "sprinting" - a metaphor I like because it emphasizes that this is fully consistent with trying to finish as fast as possible. In particular, I prefer the goal of "Make at least a bit of progress on 95% of days, and in 100% of weeks" to the goal of "Make so much progress today that it makes up for all my wasted past days." (The former goal is not easy! I think aiming for it requires a lot of interrupting oneself to make sure one isn't spiraling or going down an unproductive rabbit hole - rather than a lot of "trying to pedal to the metal," which can run right into those problems.)
It’s a bird, it’s a plane, it’s a schmoe!
This section is particularly targeted at effective altruists who feel compelled to squeeze every ounce of productivity out of themselves that they can, for moral reasons and not just personal pride. I think this attitude is dangerous, because of the way it leads people to set unrealistic and unsustainable expectations for themselves.
My take: "Whenever you catch yourself planning on being a hero, just stop. If we’re going to save the world, we’re going to do it by being schmoes." That is:
If effective altruists are going to have outsized impact on the world, I think it will be mostly thanks to the unusual questions they’re asking and the unusual goals they’re interested in, not unusual dedication/productivity/virtue. I model myself as “Basically like a hedge fund guy but doing more valuable stuff,” not as “A being capable of exceptional output or exceptional sacrifice.”
Be virtuous first!
I don't think you're going to get very far with these "vices" alone. If you aren't balancing them with the virtues of open-mindedness, self-criticality, and doing the hard work to understand things, it's easy to just lazily write down some belief you have, cite a bit of evidence that you haven't looked at carefully or considered the best counter-arguments to, and hit "publish." I think this is what the vast majority of people "investigating" important questions are doing, and if I were writing tips for the average person in the world, I'd have a very different emphasis.
For forming opinions and writing useful pieces about important topics, I think the first hurdle to clear is being determined to examine the strongest parts of both sides of an argument, understand them in detail (and with minimal trust), and write what you're finding with reasoning transparency. (All of this is much easier said than done.) But in my experience, many of the people who are strongest in these "virtues" veer too far in the virtuous direction and end up punishing themselves for missing unrealistic self-imposed deadlines on impossible self-imposed assignments. This piece has tried to give a feel for when and how to pull back, skip steps, and go easy on oneself, to make incremental progress on intimidating questions.
Footnotes
In practice, for a report that isn't claiming much rigor, this often means demonstrating “This isn’t even suggestive, it’s basically noise.” Here's a long debate about exactly that question for one of the key inputs into my views on transformative AI! ↩
My favorite real-life example of this is Barry Bonds in 2002. So many star players try to play through injuries all year long, and frame this as being a "team player." I remember Barry Bonds in 2002 taking all kinds of heat for the fact that he would sit out whenever he got even moderately injured, and would sometimes sit out games just because he felt kinda tired. But then the playoffs came around and he played every game and was out-of-this-world good, in a season that came down to the final game, at age 38. Who's the team player? ↩