One of the computer science tricks I know might be helpful here.

A depth-first search takes less memory than a breadth-first search, since you only have to remember your current path through the tree, rather than your progress through entire layers, which can get quite large.

The problem with depth-first is that you have to hit the bottom before you can go sideways at all. It doesn't work on infinite trees, or graphs with cycles. You keep going deeper, even if what you were looking for was only a few steps from the root, you may never get there. Breadth-first search is going to find the low-hanging fruit first.

But there is a best-of-both-worlds approach with the benefits of both: iterative deepening depth-first search. You create an artificial "bottom" by limiting how many steps deep you go.

Start with three, say. Once you hit your limit, you try a sibling instead of going deeper, or back up a layer if there are no siblings. This will quickly find any fruit hanging within three steps deep, like a breadth-first search would. But it will fail to find anything any deeper than your limit. If you want to search more, you increase your limit. Maybe by three if your branching factor is low, or maybe only by one if you're already deep or if it's high. Then you start over. You'll search the top of the tree again (kind of a waste), but that's the smallest part, so it's fast compared to searching the bottom at your new depth. This tradeoff is often worth it for the lower memory requirements. For a human, the repetition of the parts near the root will probably tend to shift that part into your long-term memory, making it even faster, so that's the part worth getting familiar with anyway.

Some thoughts:

• Just practice writing down all your thoughts until it stops feeling so high friction
• Alternately, practice talking out loud, as if you were explaining what you were doing and what the problem was, until that feels natural. Generally find some way of externalizing working memory.
• Specifically cultivate the trigger of "notice you're stuck in some way -> stop/drop/do-metacognition until you're no longer stuck." i.e.
  • ask yourself why you are stuck. what seems hard about this?
  • figure out a solution to the stuckness problem. 

FWIW I tend to see a good part of ADHD medication's effect as changing the trade off between exploration and exploitation. ADHD being an excess of eploration, the meds nudging towards excess of exploitation. If you struggle with a perceived excess of exploration, you might ask yourself if you are helped by taking those medication or if you might fit the diagnostic criteria.

Related : Taking too much of those psychostimulants gives usually an extreme type of exploitation often called "tunnel vision", which can be detrimental as it feels like being a robot doing something on repeat.

Also : branched thinking is not only related to ADHD but also to people with unusually large IQ. So let me just stress that YMMV

Also2 : another interesting thread about ADHD the other day : https://www.lesswrong.com/posts/zi6Qq5jeWat8xFodq/what-works-for-adhd-and-or-related-things

An option that will probably not meet your goals is to choose at the outset which solution you are going with, and value sticking to the plan more highly than you value the possibility of discovering/inventing a better solution. If you've ever taken a CPR class, this flowchart simplification was part of the curriculum. If you see someone become apneic and pulseless, you start CPR immediately rather than pondering whether or not it would be a good time to experiment with cooling their body to reduce neural damage until they can be re-perfused in a hospital environment or something. That cognitive topiary option of chopping off all but a few branches is sometimes great, but probably not appropriate to the problem-solving you're talking about.

The option that's more likely to meet your needs is to reduce the friction of note-taking. When you externalize your cognition, all the usual rules about written communication for a wider audience can be safely discarded. The purpose of taking notes when tackling a complex, branching problem is strictly to retrace your steps to an exact prior branch point, re-gather the context from that point, and continue down a different branch. Your notes do not have to make sense to anyone other than your near-future self.

When I'm personally exploring conceptual structures too big to let me just memorize my path through them, I take a linear series of note-fragments. I use more detail before and after branch-points that I anticipate wanting to revisit. When returning to a prior concept in order to branch off of it in a different direction, I simply repeat the words that are shared between the former path and the new one, and then go from them. Repetition of a short phrase is generally enough to cue me-when-rereading that it's a jump back up the tree.

Some combinations of people and problems do better with mind mapping techniques. Some combinations of people and problems do better with putting concepts on sticky notes, index cards, or other small objects, then physically moving them around. For the combination of you and your most frequent problem type, develop a note-taking practice that imposes minimum overhead while fulfilling the purpose of helping you visualize and/or navigate the concept that doesn't all fit in your brain at once.

Also, don't assume that any skill worth having will be easy the first time you try it. If you don't remember learning to type, consider the feeling of learning a new "more efficient" keyboard layout. Getting unstuck from a local maximum usually requires going downhill in some way, before things start improving again.