Third in a series of short rationality prompts.
My opening rationality move is often "What's my goal?".
It is closely followed by: "Why is this hard? And, what can I do about that?".
If you're busting out deliberate "rationality" tools (instead of running on intuition or copying your neighbors), something about your situation is probably difficult.
It's often useful to explicitly enumerate "What's hard about this?", and list the difficulties accurately, and comprehensively[1], such that if you were to successfully deal with each hard thing, it'd be easy.
Then, you have new subgoals of "figure out how to deal with each of those hard-things." And you can brainstorm solutions[2].
Sometimes, those subgoals will also be hard. Then, the thing to do is ask "okay, what's hard about this subgoal, and how to do I deal with that?"
I'll do one example that's sort of "simple" (most of what I need to do is "try at all"), and one that's more complex (I'll need to do some fairly creative thinking to make progress).
I'm reading a LessWrong post that has an equation in it. I'm not really a math guy. Normally, I just bounce off posts with equations in it. This time, the post has exactly one equation, it's otherwise an interesting post.
I could skip over the equation, but then the next paragraph is immediately referring to it and it seems kinda loadbearing.
Okay, uh, what if I actually tried reading the equation?
I try. It's hard. My eyes glaze over. A few minutes later I'm suddenly on Facebook.
What's hard about this?
Well:
Let's take the first one. Seven terms. I have ~four working memory slots and I need some of them for reasoning about the equation. Do I have some way of dealing with that?
Let's brainstorm some options:
Option 1: look at only a small part of the equation at a time.
...oh, okay, that just works. Okay, I focus on just the left side of the equation. There's an f(x) there.
What's f?
What's x?
Now we're on Hard Thing #2: I don't know what any term in the equation means. How do I deal with that? Some options:
Option A: reread the previous paragraph
...oh, that also immediately pays off. It turns out I had skimmed the previous paragraph because I knew it was talking about the equation. It explains literally what every term means. It's a bit overwhelming, but, I read one sentence at a time (reusing Option 1, above), and sit with each term until I get it before moving on to the next.
It takes a little while, but, I do it.
The next example is longer, because often, a problem is fractally hard. Dealing with that is an important life skill.
I'm trying to fill out some paperwork. It requires some information I don't know how to get, and I expect to be annoying to get.
Also, I didn't get enough sleep and am tired. My eyes are sliding off.
Default outcome: "ugh, this sucks", bouncing off, intermittently trying to work at it and then screw around on Facebook, neither successfully having fun nor completing the form.
What's hard about this? Well:
Trying to deal with both of those at once feels overwhelming. But, now it's more obvious I could split them up and deal with each separately.
#1. I'm tired
Let's start with this, since it'll be a force multiplier on the other. Let's brainstorm some options. I come up with:
I try drinking some water. I'm still tired. I try doing a couple jumping jacks. I'm still tired.
Recursion: Turns out taking a nap is Hard.
Take a nap? Ugh, that'll take awhile and I'm kinda shit at napping and I just drank 2 red bulls. I don't expect napping to work.
Hmm. What if I tried just laying down in a dark room for awhile and half-assedly pretend to nap and let my mind wander?
Well, if I did that, I'd end up playing on my phone and not even doing half-assedly napping.
What if... I didn't bring my phone into the dark room?
Okay, you know, that might just work.
I try that. It works okay – I'm not fully rested but I can focus a little more easily.
#2. I don't know where the information is
The silliest part here is how, by default, I'll periodically "start thinking about filling out the form", and try some autopilot-y move like "glance around on the tax form, hoping an answer will jump out at me." And then that fails, and I am mysteriously on facebook.
Writing out "I don't know how to get the information. How do I deal with that?", prompts the embarrassingly obvious answer of "I have to somehow figure out how to get the information" which implies "I need a plan for getting the information."
But, I notice my brain is still sliding off this problem. I try to think about it, but even with my fresh drink of water and/or nap there is something muddled and murky about it.
Recursion: Finding the information is also hard.
After thinking for a bit, I realize two things:
Hrmm.
When I direct my attention to the second thing, what do I do about that? Well, some options:
When I get to option #3, my brain viscerally rebels and screams loudly: "no, if I did rationality to solve this then I might figure out how to do the paperwork and then I'd have to do the paperwork."
Oh, well, uh, that's a problem.
It's such a hard problem that my advice for solving it is multiple entire other posts: Deliberate Grieving, and "Buckle up bucko, and get ready for multiple hard cognitive steps." In the real life case this is based on, I struggled even to do that, and I did have to find a temporary Thinking Assistant to periodically check in and remind me to focus.
This takes like an hour and is kinda embarrassing. For this blogpost I'm skipping over the details to the next simple step:
Back to problem A.
I don't know how to find the information. I need to make some kind of actual plan to find it. What are some things that might help?
This part is probably fractally hard, where each option is likely to run into another hard thing. But now that I've buckled up it feels less obnoxious to think multiple steps ahead.
Some ideas:
Which of these ends up helping depends on the exact paperwork I'm dealing with. But, it's kinda embarrassing how often I find myself "not even trying" to figure out how to find the information, and instead just sort of bopping around hoping it magically becomes easy. Once I've articulated the problem, it's usually easier to solve.
So I list out the confusing bits. I turn each confusion into an explicit step. I do the steps.
The hard problem has become kinda easy, if still kinda annoying.
And then I'm done.
You may notice there was a lot of content in example #2. In fact, so much so that I buried some of it behind multiple other blogposts. (I literally wrote "Buckle up bucko, and get ready for multiple hard cognitive steps." in the middle of writing this post, because I realized it was a whole skill to itself)
This is a fairly common feature of hard problems. It sucks. You gotta deal with it somehow.
There's a few ways of dealing with it. You can get out notebooks and try to externalize your working memory. You can tackle small pieces of a project at a time – you can maybe even bounce between two different hard problems that are hard in different ways, and let your hindbrain digest the last difficult thing from the first problem.
Since I wrote it stream of conscious style, I'm going to re-summarize the high level details:
At each step, I brainstormed multiple options.
I solved "being tired" by half-assedly napping, and leaving my phone outside my room.
I solved 2B by grieving the fact that I had to the paperwork even though it sucked, and by mentally preparing for multiple annoying steps, and getting someone to help sit with me and focus.
I solved 2A by a mix of reading the instructions more carefully, writing out explicitly all the steps I was confused by, and googling the parts I didn't understand. I turned it all into a simple checklist of instructions to myself.
The bureaucracy exercise was "easy" in some sense. You know how to fill out a form. You've done it a hundred times. Humans have done it millions of times. The only reason it's "hard" is that it kinda sucks. But if you sit down and carefully think through it, it's not too bad.
Sometimes, the reason something is hard is more like "you just really don't know how to do it", and you will need to find a new way of looking at the problem.
Recently, I vibecoding a fun side project. I wanted to make a website with lots of different versions of sheet music for a given song, and the ability to convert one type of music notation into another. One subthread of that was: can we scan static PDFs of sheet music, and convert them into a file format I could edit?
I asked a Claude Agent to write a conversion script.
It didn't work.
I said "hey Claude, it didn't work."
Claude said "oh, my bad. Let me try <some random ass thing>." Claude tried that random ass thing, and said "okay I fixed it!" It was not fixed.
I said "Please think harder, and make sure you fully understand the problem before declaring yourself done."
It did some minor ritual-of-cognition that looked vaguely like thinking harder, then did more of the same thing.
Geez Claude, you're debugging like me on a bad day, when I randomly flail instead of making a systematic checklist and trying to understand the problem.
If this were a serious project, I would say "ugh, I guess I have to debug for real", and Buckle Up again and start reading documentation and making lists of my confusions and building a model of the situation.
What's hard about this?
What's hard about that third option?
Well, it's that this task is apparently hard for Claude.
What's hard about this for Claude?
Well:
If I could solve any of those problems for Claude, it might stop claiming to be done when it clearly wasn't done, and instead keep trying.
Problem #1 does seem like a lost cause. Someday AI will have good judgment to navigate complex domains when it's first patternmatch guess doesn't work. Today is not that day. (And when that day comes, I'm scared).
Problems #2 and #3 both mean that Claude just has no ability to tell if it's done. Maybe I could convince it to stop pretending it was done, but that wouldn't get it done any faster.
I feel really suspicious of #2. Claude seems like it should be capable of reading through a codebase line-by-line, including stepping into the foreign library, and modeling whether each step should work. I've tried a whole bunch to get it to do this reliably, and haven't gotten it working yet. Maybe I'll find a good metacognition prompt someday, but, having put a dozen hours into it, it's not looking promising tonight.
#3?
Why can't Claude look at my website, click buttons, and see if the conversion is working?
There's actually some more recent tools that let it do directly that. I haven't bothered to install those tools because I'm lazy. I could install those tools, but then I'd have to do a few annoying steps in a row to make sure the tools work. This is a fun side project. I don't feel like it.
Hmm. Why does Claude need to navigate a webpage and click a button?
Because currently, the conversion script is built directly into the webpage, which can only be accessed via a browser.
Is there any good reason for that?
No. I could refactor this into a script that can be run individually as a test case, from the command line. The Claude Agent does have the ability to make and run scripts on the command line (if I give the permissions to do so).
"Hey Claude, can you write yourself a script that tests the conversion function, and continue re-running it while debugging until the script reliably works?"
"No problem." Claude does that. A few minutes of brute-force exploration later, it has solved the problem. It turns out, the debugging principle of "first, try to cleanly replicate the problem" is just as important for LLMs as for humans.
(This was only a week ago, but it seems generally useful for getting AIs to keep working until they're actually done. Sometimes it's a bit harder to convert a problem into something the AI can "see", but, I now have in my toolkit "think about how to convert a coding problem into something the AI can see")
When you have a task that feels frustratingly difficult, try asking yourself "what's hard about this?", and then followup with, "what do I do about that?"
Sometimes, this is much easier than these examples, and as soon as you ask "why is this hard?" you'll realize "because I need to think for 2 goddamn seconds in a row and I wasn't bothering to do that", and then you think about it for 2 goddamn seconds in a row and then the solution is obvious.
Sometimes, dealing with the Hard Part is also separately hard, and you need to recursively ask "Okay, why is that hard? What can I do about that?"
What is a difficult thing you're dealing with lately? What's hard about it? How can you deal with that?
(If you're comfortable writing up your answers in the comments, I'd appreciate it)
(Well, for lighter-weight problems, it may be enough just to list the single-hardest thing if it easily occurs to you)
See The Babble Challenge and Metastrategic Brainstorming for some background on training the muscle of "generate approaches to solving problems." (Though note one commenter's reply "sometimes the solution isn't actually to 'brainstorm', it's to go on a walk and let the my mind wander until it figures it out.")