CGP Grey describes a phenomenon I think of as 'dreams of ideas' that I find useful as a tool to know when to stop working on a project.

You can be working on something and you are thinking about what it could be, but what is hard to know is if that thought in your head is more like when your brain dreams.

There is a phenomenon that I always think of as like, you wake up and you think you had a great idea in your dream, but what you really had was the idea of a great idea. You didn't have a great idea, but you had all the emotional resonance to it being a good idea in your dream. You're like, "Oh, why can't I think of the details?". It's like, because there weren't any details there at all...

Sometimes there just isn't the better version of the thing. Like, that's not real. You can't get the details right because those details don't exist. You're having some kind of different response to the thing that you're working on, which is, "This thing isn't good. I can imagine a better version of it." But can you really? Because, like, the details are what matter.

I sometimes get an idea for a good story and feel inspired to try writing some fiction. Then I sit down to write and realize that I don't really know what my idea was beyond a vague image of one tiny slice of a story.

Sometimes I think of awesome research ideas and feel excited to start a project. Then I sit down to start coding and realize that I can't implement it because my idea wasn't really complete or coherent. I had a vibe of the sort of the thing I'd like to have done, but not an actual plan for what to do.

When to cull ideas

This sort of sounds like the opposite advice to Butterfly Ideas but it's actually not in contradiction. When an idea is first born, it should be nurtured and protected from excessive rigor. But it's not healthy for it to remain in that state for too long. You need to set a 5 minute timer and think hard about the thing to see if you can find the details to make it work. Later in the lifecycle, once it's had time to develop, if an idea hasn't matured, then it's time to start asking if this is a real idea or just a mirage.

Examples

I think it's particularly useful to think about this very late in the lifecycle of an idea. Once dozens of people have thought about something for months or years, then there should be very clear details in place, or very good excuses for why the idea isn't concrete yet.

It's obviously problematic to characterize a whole academic discipline with a broad and dismissive brush. And I do genuinely think there are good and concrete ideas to be found there. But the study of complex systems seems to some extent to be a dream of an idea. There is a lot of appeal around the vagaries of explaining emergent phenomena but the technical methods employed are often underwhelming compared to this vision. For example, simulations, especially simulations of many individual agents (LLM or otherwise) are exciting but have delivered relatively few important discoveries.

B2B companies are particularly vulnerable to becoming enamored with dreams of ideas because they only need to sell the dream to another company to make money (at least at first). A software company I once worked for hired a new team to build a 'knowledge graph' product. Whenever I spoke to the team I was always left a little unsatisfied by their answer to the question "what is this product and how is it useful?" A knowledge graph is basically a type of a database. And sure, databases are useful, but somehow 'Knowledge Graph' evokes much greater expectations, even in the people making them.

Early stage startups will also often maintain dreams of ideas for too long. Many AI startups have big dreams but soon realize that the best product they can make is a simple LLM API wrapper. Sometimes that creates exciting new value, but other times it ends up being less useful than the LLM it is wrapping.

Policy documents often include calls for 'innovation'. Of course innovation is a real thing and we should make policies that enable it. But a call for innovation itself is almost the Platonic ideal of a dream of an idea. It's the idea to have better ideas. Which is generally not a very useful idea.

Not every unfulfilled idea is a dream

Mechanistic interpretability has had few big wins, but I don't think it's a dream of an idea. To me, reverse engineering a neural network means being able to (at least in theory) hand write a normal human understandable computer program which performs the same function as the network. And there are concrete techniques like SAEs or Activating Patching that people have tried to apply to this problem.

The unified theory of physics is yet to be found, but may well be some day. The point being that, just because an idea hasn't been fully solved, doesn't mean it will always be a dream.

The solution

Luckily it's quite straightforward to mitigate this thought disease in your own life. Any time you have an exciting idea, write it down as quickly as possible, including all the detail. (This is good for many other reasons as well - like not forgetting your ideas). The back of your mind is the warm hospitable environment in which soothing dreams of ideas can fester and steal resources from real, useful thoughts. By exorcising the thought out of your daydreams and down onto paper, you can quickly expose it as vacuous hopium if that's all it is.

This blog post, on the other hand, was the result of quickly writing down a thought and deciding that it was actually worthwhile.

TL;DR

A "dream of an idea" is an idea that you think is exciting or valuable but lacks the concrete details that make a "real idea" useful. People often dream of ideas for too long because it is more comfortable than facing the harsh reality of implementation.