Epistemic status: very confident
See also: A closely related post by Gwern, another related post by John, and some interesting slides from a hacker's talk. None of the concepts here are new, but I've tried to lay them out in a more helpful frame.
When good programmers debug hard problems fast, it's usually because they understand the system well enough to track the important internal state in their head, letting them drastically reduce the solution space they're searching over.
This post contains my advice from ~5yrs of linux & programming experience on one of the primary ways to getting better at programming: white-boxing.
Definition and clarification
Definition: White-boxing, the process of taking a system you reason about purely in terms of input/output abstractions ("Autograd takes code and outputs gradients") into a system who's gears you understand ("Autograd takes code, records operations to construct a computational graph, then computes gradients via the chain rule")
There are three important things to understand about white-boxing:
First, White-boxing goes through various shades of gray. When you hit diminishing returns you want to switch to understanding another system. (Though if you find a topic fascinating then go ahead and do a deep dive!)
Second, It is higher value to white-box leaky abstractions. Autograd for ML is a great example of a leaky abstraction, if you mix up permute and view your gradients can be subtly wrong. See Karpathy's great post for more on this. On the other hand, the CPU is a very good abstraction, unless you're doing something unfathomably cursed, you should never run into CPU bugs.
Third, and perhaps most important for building skill,[1] you must notice when you're going into brute-force search mode, and then take action by investing time in understanding the underlying system, until both the problem and solution make sense.
Absorbing the pattern
Read Gwern's list and then attempt to come up with three new examples of the pattern, ala framing exercises. I used to think I had absorbed the concept, but I was still black-boxing things without realizing it. I encourage the reader do another exercise: Come up with three examples of systems (preferably computer systems) that you've recently been partially black-boxing, and problems you ran into because of this. Alternatively, come up with examples of you doing black-box search, and how inefficient this was. Try and install the trigger-action-plan for "notice black-box search, understand things instead"
It may seem I'm making a big deal of this, but it is critical to notice when you don't understand something, and then take action by understanding it, making a note for later, or something else. Not doing this has caused me to unintentionally plateau for years at certain things (like CSS).
Conclusion
Go out there and understand systems! Watch talks, read articles, reimplement existing software. We built computers, a human wrote every line of code that's being executed. You can understand it.
Notice when you're doing brute-force search due to a lack of understanding, and take action to build that understanding. The investment will pay off, often immediately, as a black-box search for solutions can be extremely inefficient.[2]
Finally, here's a Github megalist of resources around "building your own x" - one of the best ways to understand a system is to build it yourself, so go out there do that! open the black box!
