In the last month and a half, I've had more (of what I believe to be) profound, creative insights to technical problems than in the five years prior. For example, I independently came up with the core insight behind DenseNets during the second lecture on convolutional neural nets in my Deep Learning class. I've noticed that these insights occur as byproducts of good processes and having a conducive mindset and physiology at that point in time. I'm going to focus on the former in this post.
End in Mind
Often, when I'm confused about a technical problem, I realize that I don't even know what a solution would look like. Sure, I could verify a solution if one were handed to me on a platter, but in these situations, I generally lack a detailed mental model of how I'd want a solution to behave. This may sound rather obvious once brought to your attention, but I'd like to emphasize how easy it is to spend hours in murky, muddled, pattern-matching gradient descent, mindlessly tweaking the first ideas that came to you.
Incredibly, generating a detailed solution model is often enough to quickly resolve (or at least reduce) even formidable-seeming problems.
The aforementioned process is backward chaining (as opposed to forward chaining, which is what you're doing with the work-from-what-I-have-now approach). Take the time to actually visualize what a solution to the problem would look like. How would it behave in response to different inputs? Are you offloading all of the work onto a weasel word (like emergence)?
When you're done, map-territory confusions should be ironed out, it should be clear whether current approaches truly tackle the core issue or just something similar-looking, and black-box terms should be disintegrated. If you can't do this, you probably can't solve the problem yet - you should spend more time working on your understanding. I encourage you to read this multiple times - the bulk of this post's value resides in this section.
For example, yesterday I thought about a smaller problem for about 15 minutes, and had nothing to show for it. I almost resigned to asking a more mathematically-experienced friend if he had any ideas, but instead forced myself to go through the above process. The answer was immediately obvious.
In my experience, proper solution formulations are accompanied by a satisfying mental 'click'; this is when you eagerly chase down all the freshly-exposed avenues of attack. Push the frontier of new understanding as far as it will go, repeat the process when you notice your thought process again becoming muddled, think with paper if the problem is complicated, and enjoy the rush.
Qualia of Discovery
I can't impart to you the exact feeling of using this technique properly, but I can make its fruits concrete by selecting a few things I've dissolved recently:
- I've observed samples from a normal distribution and recorded the mean and variance. New observations are going to come in, and I want to be able to assign some scalar quantifying how unusual this looks for the distribution.
- Challenge: I have not yet had the opportunity to thoroughly study a statistics textbook. How can I produce a solution without having to post this on StackExchange or guess the name of the actual technique?
- Answer: Srrq gur fgnaqneq qrivngvba bs gur arj bofreingvba vagb gur gnau npgvingvba shapgvba gb trg n qvssreragvnoyr zrnfher bs qvfgnapr-sebz-qvfgevohgvba.
- A binary classifier has observed a bunch of 'dog' pictures, and we want it to be able to robustly communicate when it sees something new - knowing what it don't know.
- Challenge: How can we robustly represent the 'unknown' label in classification?
- Answer: Purpx bhg zl cbfg ba nzovthvgl qrgrpgvba sbe n cbgragvny nafjre.
- I'm a reinforcement-learning agent named R̸̵̕Ȩ͟Ḑ́A͝͏̢C̷͜T̸҉E͘D̶͘͟, and I'd love to fetch things from other parts of your house for you, as quickly as possible.
- Challenge: How can we robustly disincentivize agents from messing up the environment while optimizing their objective?
- Answer: I won't even ROT13 this one - I'm not sharing this until I've polished my work more. Attempting to deduce my response is left as an exercise to the reader; I promise it doesn't require more than 96 insights in series.
Look out for my review of Artificial Intelligence: A Modern Approach, coming soon!