It’s not the only thread I’m pulling on.

I think this is worth expanding on- in practice, I've found the strongest method for avoiding the "oh no my great idea is not working out but I'm stuck in it" trap is to have other promising options just waiting for you to poke them.

Instead of feeling trapped and entering a cycle of motivation-killing burnout, a dying idea starts feeling just... kind of boring, and you naturally want to do the other more interesting thing. You don't even have to try, you just find yourself thinking about it in the shower. When refined and used intentionally, it can align productivity with natural inclinations.

I tend to organize my work as a few different parallel depth first searches. In a given day, I'll often work on two different things, and in a given week I might hit three or five. Something like spaced repetition applied to fields of study, with time for the other ideaspaces to lie fallow. This tends to produce recursively expanding options for research. After doing this for years, it's now completely impossible for me to actually investigate everything I've written down in a todo somewhere, but at least I definitely don't feel married to any one idea!

I suspect getting the full benefit does require a few (dozen) iterations of failure. Noticing that you already know an idea is dead is a lot easier when you've felt that same thing 20 times.

Emotionally Detatch yourself from your Ideas

Here is a model of mine, that seems related.

[Edit: Add Epistemic status]
Epistemic status: I have used this successfully in the past and found it helpful. It is relatively easy to do. $\frac{\text{utility}}{\text{time\_investment}}$ is large for me.

I think it is helpful to be able to emotionally detach yourself from your ideas. There is an implicit "concept of I" in our minds. When somebody criticizes this "concept of I", it is painful. If somebody says "You suck", that hurts.

There is an implicit assumption in the mind that this concept of "I" is eternal. This has the effect, that when somebody says "You suck", it is actually more like they say "You sucked in the past, you suck now, and you will suck, always and ever".

In order to emotionally detach yourself from your ideas, you need to sever the links in your mind, between your ideas and this "concept of I". You need to see an idea as an object that is not related to you. Don't see it as "your idea", but just as an idea.

It might help to imagine that there is an idea-generation machine in your brain. That machine makes ideas magically appear in your perception as thoughts. Normally when somebody says "Your idea is dumb", you feel hurt. But now we can translate "Your idea is dumb" to "There is idea-generating machinery in my brain. This machinery has produced some output. Somebody says this output is dumb".

Instead of feeling hurt, you can think "Hmm, the idea-generating machinery in my brain produced an idea that this person thinks is bad. Well maybe they don't understand my idea yet, and they criticize their idea of my idea, and not actually my idea. How can I make them understand?" This thought is a lot harder to have while being busy feeling hurt.

Or "Hmm, this person that I think is very competent thinks this idea is bad, and after thinking about it I agree that this idea is bad. Now how can I change the idea-generating machinery in my brain, such that in the future I will have better ideas?" That thought is a lot harder to have when you think that you yourself are the problem. What is that even supposed to mean that you yourself are the problem? This might not be a meaningful statement, but it is the default interpretation when somebody criticizes you.

The basic idea here is, to frame everything without any reference to yourself. It is not me producing a bad plan, but some mechanism that I just happened to observe the output of. In my experience, this not only helps alleviate pain but also makes you think thoughts that are more useful.

The tension seems to come from the idea either working or not working. Supposedly if the idea does not work there is huge sunk cost with no payoff and others that have not latched on an idea for long are forward.

If one were proud about the exploration that could successfully end in a negative result. Existence proofs are nice but impossibility proofs are nothing to sneeze at. If everybody wants to p-hack their media-sexy idea into a paper and nobody wants to do replications, on research community level there can also be issues about setting incentives of impressiveness vs research.

In the process of making a blog about an idea that ultimately went nowhere, the entertainment value of reading the blog is still around. It all does not need to hang on that one nail. Maybe the real treasure was the researchers we made along the way?

Curated. This post adds two important pieces to the "how to make intellectual progress faster on problems we don't understand" toolkit. First: knowing when to pivot away from an idea can save you a lot of time on your journey to finding the Actually Good Ideas. And second: by making it easier to look at ways your idea might fail, you're able to more quickly iterate and rework a flawed idea into something that works.

That tiny note of accomplishment when John Wentworth links the selection theorems tag I created in one of his posts. 😌

It's a very little thing, but it's useful to me and it's nice to know that it's useful to senpai as well. 🥺

This was thorough, concise, and helpful. Thank you.

So, I have an internal sense that I have overcome "idea scarcity", as a result of systematized creativity practice (mostly related to TRIZ), and I have a suspicion that this is both learnable and useful (as a complement to the domain-specific approach of "read a lot about the SOTA of alignment"), but I don't know how useful; do you have a sense that this particular problem is a bottleneck in alignment?

I can imagine a few ways this might be the case:

• Junior researchers come up with one great idea and then burn out (where they might have been able to come up with 2 or 3 otherwise); most researchers are junior in such a new field, so fixing this would nearly double or triple the number of great ideas, increasing the chance that one of them succeeds (plus positive second-order effects).
• Researchers waste effort working on an idea after it's no longer promising because they worry they won't come up with a new one (where without the fear, they would have shifted back to "explore" sooner); back-of-the-envelope, I imagine this would save about 10% of researcher time (again, with positive second-order effects).
• As a field, we're doing too little work to find fatal flaws in ideas, in part because there is a shortage and it would be too demotivating, which leads to a similar dynamic as above, where execution effort is spent on ideas that should have been shelved.

… that wasn’t enough to learn the pattern, though. Shortly out of college, reality was still hitting me over the head; that time the big idea was an efficient implementation of universal competitively-optimal portfolios. I lost a couple thousand dollars on wildly over-leveraged forex positions.

I am curious what that idea was and where it went wrong. 

I liked this for the idea that fear of scarcity can drive "unreasonable" behaviors. This helps me better understand why others may behave in "undesirable" ways and provides a more productive way of addressing the problem than blaming them for being e.g. selfish. This also provides a more enjoyable way of changing my behaviors. Instead of being annoyed with myself for e.g. being too scared to talk to people, I look out for tiny accomplishments (e.g. speaking up when someone got my order wrong) and the benefits it brings (e.g. getting what I wanted to order), to show myself that I am capable. The more capable I feel, the less afraid I am of the world.

I'd say it depends on the situation as well as on the idea.

If you are unable to test it for a while but feel reasonably sure you will be able to do it well, given the chance;

If at the moment, you are, demographically, someone who is likely to stop pursuing ideas regardless of what they amount to, properly tested;

If you have obligations (or expectations, yours or not) which you can meet more easily following your idea whether it works or not;

If you do have other ideas, and they are cool and everything but would require more work and - worse - are much less well-defined than your Big Idea;

If you wouldn't have to depend on others (much) to follow exactly this one idea and you don't feel like those others are really interested in working with you after all;

If your field of study (or whatever) is just built around... well, data... and there have been a few people you admire who went with their ideas instead of simply collecting observations...

then yeah, idea scarcity bites hard.

So can you control emotion with rationality, or can't you?  "There's more fish in the sea" seems like classic emotion response control.  Or maybe it's that "emotion" vs. "feelings" idea— one you have control of, and one you do not?  Or it's the reaction you can control, not the emotion itself?

Having to "take a dream out behind the woodshed", as it were, is part of becoming a whole person I guess, but it's, basically by definition, not a pleasant experience.  I reckon that's by design, as sometimes, reality surprises you.

I think it boils down to the inherent paradox of persistence.  There are adages about both ends of it— i.e. giving up too soon, and not giving up soon enough— and neither is "wrong" per se.  I think mainly it can be hard to tell which is which, and maybe instead of looking at things as win or lose or pass or fail, we should, as someone already mentioned, enjoy the ride.

Does being able to do judo on our emotions count as being able to control them?  Is this all semantics?  I dunno— but I'm glad you found something that works for you, and share it in the hope that it helps others.

A good methodology for idea-generation, which avoids attachment to the "but-it's-my" idea:

1. Explore the design-space. Look at all the possibilities; because it would be ignorant to ignore them. You aren't supposed to find 'the best' here; this is about answering "What is Possible? What are my Options?" If you've only come-up with three to five (e.g. "It has to be Capitalism, Communism, or Socialism") then you haven't even started. I looked at a few dozens of ways to combine Zeppelins, Paramotors, and Sail Boats... five options ain't even started.
2. Look at the pro's and con's of each idea - in order to find the patterns between those ideas. You're not picking 'the best idea'! If two ideas both share a particular failure, that's where you ask: "What is it about these designs that leads to that failure?" Now, you have a better grasp of what sorts of problems come from what kinds of designs. No good designs, yet!
3. Pick lots of weird niche-application-spaces. NOT because you plan to market into those niches - no! Rather, when you ask "How would I have to design this for an Arctic Marine Environment? Or, if it has to fit in an elevator, and fold into the trunk of your car?" You are narrowing-down your options in that vast design space, creating extra constraints, more problems to overcome beyond the normal. That is when you have to hunt for the weird and surprising solutions. Stack those odd-balls up! None of them are good stand-alone designs.
4. You now have a good sense of what's possible, and what designs create which sorts of problems, and you've come-up with lots of awkward niche-solutions. Now, you can glom those niche-bits together, and see if any of them stick; this is combinatorial search, so you're probably best meandering for a while. If anything stands-out, see how you can hone-in on it. Most of the time, you're still left with a pile of junk! Sometimes, you have a new algorithm, or a new touchscreen, or a zeppelin-paramotor-caravel.

So, by shuffling through and clustering solutions according to which problems they share, (step 2) then you are actively identifying all the problems, in general... including the problems with your 'favorite,' early, almost-certainly-bad idea.

"mistrust" is not the best approach here. Mistrusting yourself or your ideas can lead to misery and feeling lackadaisical. Could lead to lacking motivation to pursue an idea as hard as you otherwise might.

"Openness" is a better idea imo. Openness to the idea failing or taking some adjustment to reach success, openness to it succeeding as well. Looking at ideas not just as a way to achieve success, but test the view you have on the world, a way to learn something new about the world through testing it and working on it.