Exploring the Idea Space Efficiently

by Elithrion4 min read8th Apr 201227 comments


Personal Blog

Simon is writing a calculus textbook. Since there are a lot of textbooks on the market, he wants to make his distinctive by including a lot of original examples. To do this, he decides to first check what sorts of examples are in some of the other books, and then make sure to avoid those. Unfortunately, after skimming through several other books, he finds himself completely unable to think of original examples—his mind keeps returning to the examples he's just read instead of coming up with new ones.

What he's experiencing here is another aspect of priming or anchoring. The way it appears to happen in my brain is that it decides to anchor on the examples it's already seen and explore the idea-space from there, moving from an idea only to ideas that are closely related to it (similarly to a depth-first search)

At first, this search strategy might not seem so bad—in fact, it's ideal if there is one best solution and the closer you get to it the better. For example, if you were shooting arrows at a target, all you'd need to consider is how close to the center you can hit. Where we run into problems, however, is trying to come up with multiple solutions (such as multiple examples of the applications of calculus), or trying to come up with the best solution when there are many plausible solutions. In these cases, our brain's default search algorithm will often grab the first idea it can think of and try to refine it, even if what we really need is a completely different idea.

Of course, the brain is not so stupid that it will choose the first idea it has and refine it forever. Even in the ancestral environment, we would have run into problems where depth-first search is not very effective. Rather than spending time on refining your skills at chipping flint to make tools, for example, you may have been better served by learning to pick a better type of flint to work with before even starting. In modern times, however, these problems have grown more challenging and more numerous. I'm sure all of us have had the experience of working on some problem for a long time, refining our solution, maybe even trying to make the problem fit the solution we came up with out of frustration, only to give up, come back later, and then suddenly have a completely different and obvious solution come to mind. While part of this is probably due to some peripheral processes in our brain analysing the problem while our conscious thoughts were not focused on it, I think the key component is that by leaving the problem alone we "forgot" our first solution and were free to look for a better one.

Similarly, it is possible that if Simon stops trying to come up with examples whenever he remembers the examples he's seen before, and only returns to the task when his mind is relatively blank, he might be able to produce something original. Even if he does that, unfortunately, he might still find himself coming up with only one example at a time, and then being stuck thinking only of examples that are somehow similar to it. Either way, having to pause for several hours every time he finds himself primed to think of something is far from ideal, and there are better solutions.


The fundamental thing you should do when approaching a difficult, but tractable1, problem is to avoid proposing a solution immediately. The moment you propose a solution, your brain will be primed to to try to refine it or to look for similar solutions, even when it might be much more efficient to further analyse the problem or to look for other, radically different, solutions. Even when you have thoroughly understood the problem, however, you should still wait before proposing a solution, unless the problem is fairly easy and one solution is all you need.

What I would recommend that you do before that is come up with a map of the idea-space, describing where the possible solutions might be found. For instance, before looking at a single example of calculus, Simon might have written down a list of idea areas to explore: "jobs, personal life, the natural world, engineering, other". He would then take the first broad category, "jobs" and expand it into a longer list, perhaps "agriculture, teaching, customer service, manufacturing, research, IT, other". With this longer list, he can then focus on each area in turn and either expand it further if it seems especially rife with examples, or come up with an example from the area directly. Once the area is depleted, which he might decide is the case if it takes him longer than one minute to come up with an example, he would move on to the next area.

There are two main advantages to this approach. The examples Simon finds should be fairly representative of all the examples he can think of, since he started with a map of all such examples, and, better yet, he should be able to find examples much faster because he knows to stop looking in one small area when it becomes depleted.

The same approach is also useful when you're trying to come up with the single best solution. For example, if you're trying to come up with a way to deal with climate change, you might write down "reduce carbon emissions, engineer the climate to be better, adapt to climate change, other" and move on from exploring one option to the next when the option runs into significant difficulties. Note, however, that in this case it is important to arrange your options in order of how likely you think you are to find your best solution within each of them to make sure you explore the most solution-rich areas first.

In general, there are three main things2 to keep in mind when creating a map of your idea-space:

  • Don't get overly specific with the initial areas, since you will refine them when you're expanding them.
  • Try to include all the areas that might contain a solution and none that do not.
  • Try to pick areas so each of them is equally likely to contain a solution (or order the areas by the number of solutions and move on more quickly from solution-poor areas).

For instance, when Simon came up with his list of areas for calculus examples, he correctly did not include "philosophy" in the list, since it contains much fewer examples than any of the other areas.


In sum, when dealing with a challenging problem or coming with a lot of examples: don't jump to a solution, instead carefully consider the problem, come up with a map of areas of ideas where solutions might be found, and search the map until you get the solution you're looking for. If you use this approach, solutions should come to you faster, be needlessly complex less often, and be a lot more correct than the ones from your brain's naïve search algorithm.



1: It's important to note that if the problem is not tractable for you, in the sense that you can't tell if you're getting closer to a solution or not, these recommendations won't do much. For example, if I asked you "What is the next number in the sequence 14, 15, 16, 17, 21, 23, 30, 33?" it will help only slightly to hold off on coming up with solutions, and your best bet might be to start doing a depth-first search (as long as you keep in mind that you should not look for overly complicated solutions).

2: This is somewhat similar to Vladimir_Nesov's post recommending that we consider representative data sets (in particular the three mistakes he lists, which are well worth reading).

Personal Blog