A potential way to think about this:
I learn a lot from most projects, and I think this is a huge benefit from at least starting a project. I then have whatever I learned to use in the projects I do end up taking to completion.
Also remember that most projects are likely to be failures in some way (at least I think this is the case, I don't have the data lined up to back this up). Once you've squeezed much of the learning potential out of them, then it's likely you're not losing much, if anything by dropping the project.
Find some overarching theme to the ideas you are most excited about. Make your ideas into a roadmap towards whatever the ultimate achievement of the overarching theme would be. Then take the smallest possible step along that road. And then the next.
Forget about "high expected-value projects". Think about what you want to be able to do long-term. Pick projects that allow you to gather the experience and aquire the skills to do this. You motivation shouldn't be "if this project were completed it would be really cool" - it should be "everytime I work on this I gain skills/knowledge/connections/... that will lead to me being able to long-term complete many such projects".
Okay, let's do that backwards planning exercise.
In the long run, I want to do my research but live a low stress and financially comfortable lifestyle. The traditional academic path won't achieve that because I will end up doing my research but leading a high-stress and financially fraught lifestyle. There are three possible solutions to the problem, in rough order of preference A Pick a research agenda that is lucrative, so that I can supplement my income with lucrative consulting gigs and have a strong exit option B Learn to code and get a data science job, then do my research as a hobby C Get a government job related to my field (intelligence or aid)
Path A seems like the best one for both personal and EA reasons. Right now I split my time between writing on foreign investment and cabinet formation. But only the foreign investment work might pay the bills, the cabinet work ends with me in the brutal academia rat race. However, the foreign investment research might or might not succeed depending on contextual factors like competition, my ability to build a brand and the value of academic prestige in the field. So I should first try and figure out if the investment-academia path is satisfying.
I want to find out if that works over the next 6 months or so while in my academic program.
If the returns are too small and the competition too stressful, I should pivot toward a programming career. It's a well-payed 40-hour industry, and I can do my research as a hobby for 8 hours a week. That sounds like a lovely life too. So if I pick that, I would deemphasize my research and focus on coding skills for interviews and building career capital there.
I'm satisfied with that plan. The next question is, how do I stick to it? More on this later.
I also have this experience. For me the answer is to work with other people. In those weird personality test things that businesses sometimes do their is a category of person (I think, for some reason, called a "plant") who flits from one exciting idea to another and never finishes things. The silly business teamwork talk thing suggested that a good team has someone like that, then someone who is a bit more negative who can try and shoot ideas down to make sure bad ones don't get through, then finally a person (possibly the second person again) who is dedicated to making sure existing things get finished and pushes for that.
In my own experience, my personal projects flit all over the place. But in team situations I am forced to commit.
I’m glad you posted this — this may be happening to me and now I’ve read about sunken cost faith counterfactually
You mention paper ides, so I will answer assuming this is about research.
In my experience, picking a good research topic up front is extremely important, since actually testing your idea in depth will take a lot of time and there's a significant chance that once you do that you'll have to discard it anyway. So I think a period of flipping through project ideas without long-term commitment to any of them is pretty helpful for feeling out whether a given idea is worth the time and risk. Just make sure that you really seriously consider each idea you're trying out rather than just dropping it for something else, and write down what you're thinking in case you want to revisit the idea later.
During this trial period for an idea, I'd suggest biasing towards feeling out what the project might look like in the longer term rather than diving directly into implementation, though doing some initial prototyping can help you learn quickly here. Think about potential impact, feasibility given your time and resources, any unknowns you think could sink the project (investigate these ASAP), anyone you might want to collaborate with, and how it might fit into a broader research landscape. Talk the idea over with other people. Try to learn quickly here, and move on freely. Understand that if you commit to a project, there will likely be months of work before any payoff.
I have also heard the advice to keep multiple projects going simultaneously, although I haven't tried this myself. I would caution against using this as an excuse to backburner a bunch of things rather than focusing down and killing what isn't working, though.
One possible explanation+solution from the "Hammertime" author: https://www.lesswrong.com/posts/o8EWXypYGwJ6JYpBT/hammertime-day-8-sunk-cost-faith
I try to budget some time/energy for new ideas, new projects, etc. So that I can satisfy my unsatiable hunger for "new" without feeling guilty about it, but at the same time can make progress on some longer term project with the majority of my time/energy.
Nice. You might have solved the problem (will see if I actually like using it) where making a tool like this was on my list to prioritize my list, but I never got to it because the system did not exist.
The implementation is probably very inefficient and I suffered from my own scrope creepiness but the idea is still there.
Posting this comment motivated me to fix some bugs in the next few days so if you run into issues give it another go next week btw
Lately I have been starting a ton of projects, but dropping them once I lose interest. I come up with a cool paper idea, but next week I have new idea I'm excited about. Almost every idea I have requires extensive commitment to see return, so the short projects are wasted.
Does anyone else have this experience? What strategies did you use to pick the high expected-value projects and stay with them?