This post is probably right that illegible skills rely on tracking non-obvious bits of information. But I don't think that discovering that info is as simple as asking "What Are You Tracking in Your Head". Remember that there's a lot of inferential distance between the you and an expert, and they've likely forgotten all that you don't know.
Thankfully the problem of getting tacit knowledge out of someone has a growing literature on it that is quite useful. The field of Naturalistic Decision Making developed some techniques to do this, one of which is fairly simple. It is called Applied Cognitive Task Analysis. Here's a summary of it from CommonCog [1]:
There are four techniques in ACTA, and all of them are pretty straightforward to put to practice:
- You start by creating a task diagram. A task diagram gives you a broad overview of the task in question and identifies the difficult cognitive elements. You'll want to do this at the beginning, because you'll want to know which parts of the task are worth focusing on.
- You do a knowledge audit. A knowledge audit is an interview that identifies all the ways in which expertise is used in a domain, and provides examples based on actual experience.
- You do a simulation interview. The simulation interview allows you to better understand an expert’s cognitive processes within the context of an single incident (e.g. a firefighter arrives at the scene of a fire; a programmer is handed an initial specification). This allows you to extract cognitive processes that are difficult to get at using a knowledge audit, such as situational assessment, and how such changing events impacts subsequent courses of action.
- You create a cognitive demands table. After conducting ACTA interviews with multiple experts, you create something called a ‘cognitive demands table’ which synthesises all that you’ve uncovered in the previous three steps. This becomes the primary output of the ACTA process, and the main artefact you’ll use when you apply your findings to course design or to systems design.
The blog post goes in depth on this method, the theory that undergirds it and how to notice and acquire the perception that experts possess.
As to your actual question, I guess I'd say that the same holds true for video games. If you want to beat a difficult boss, then try to gather info first on what the timing is like, what cues there are for attacks and so forth.
Another area is when doing QFT calculations, you need to keep track of the interaction terms in the lagrangian and the free field terms in order to turn the time translation operator into a series of Feynman diagrams without ever bother to expand out the power series and use Wick's theorem and whatever. Makes writing out scattering amplitudes less of a chore.
Also, Feynman's trick applies to reading near anything. Keep an example in your head and see if it matches what the text says about it. Most of the time when I get confused by a text, doing this will clear things up.
P.S.
This is unrelated to your post, but if you could
choose anyone to work on AI alignment, who'd you pick?
A fantastic blog that is concerned with applied rationality. It outlines how to find and acquire tacit knowledge. I'd recommend starting from the Tacit Knowledge Series.
For pre-paradigmatic fields, I agree that the insights you extract have a good chance of not being useful. But if you some people who are talking past each other because they can't understand each others viewpoints, then I would expect this sort of thing to help make both groups legible to one another. Which is certainly true of the AI safety field. And communicating each other's models is precisely what is advocating now, and by the looks of it, not much progress has been made.
To me, it is pretty plausible that Yudkowsky's purported knowledge is tac... (read more)
A large chunk - plausibly the majority - of real-world expertise seems to be in the form of illegible skills: skills/knowledge which are hard to transmit by direct explanation. They’re not necessarily things which a teacher would even notice enough to consider important - just background skills or knowledge which is so ingrained that it becomes invisible.
I’ve recently noticed a certain common type of illegible skill which I think might account for the majority of illegible-skill-value across a wide variety of domains.
Here are a few examples of the type of skill I have in mind:
The common pattern among all these is that, while performing a task, the expert tracks some extra information/estimate in their head. Usually the extra information is an estimate of some not-directly-observed aspect of the system of interest. From outside, watching the expert work, that extra tracking is largely invisible; the expert may not even be aware of it themselves. Rarely are these mental tracking skills explicitly taught. And yet, based on personal experience, each of these is a central piece of performing the task well - arguably the central piece, in most cases.
Let’s assume that this sort of extra-information-tracking is, indeed, the main component of illegible-skill-value across a wide variety of domains. (I won’t defend that claim much; this post is about highlighting and exploring the hypothesis, not proving it.) What strategies does this suggest for learning, teaching, and self-improvement? What else does it suggest about the world?
Pay Attention To Extra Information Tracking
A lot of people have heard Feynman’s “hairy green ball thing” quote. It probably sounds like a maybe-useful technique to practice, but not obviously more valuable than any of a dozen other things.
The hypothesis that extra-information-tracking is the main component of illegible-skill-value shines a giant spotlight on things like Feynman’s examples technique. It suggests that a good comparison point for the value of tracking a prototypical example while reading/writing math is, for instance, the value of tracking the probable contents of opponents’ hands while playing poker.
More generally: my guess is that most people reading this post looked at the list of examples, noticed a few familiar cases, and thought “Oh yeah, I do that! And it is indeed super important!”. On the other hand, I’d also guess that most people also saw some unfamiliar cases, and thought “Yeah, I’ve heard people suggest that before, and it sounds vaguely useful, but I don’t know if it’s that huge a value-add.”.
The first and most important takeaway from this post is the hypothesis that the unfamiliar examples are about as important to their use-cases as the familiar examples. Take a look at those unfamiliar examples, and imagine that they’re as important to their use-cases as the examples you already use.
Ask “What Are You Tracking In Your Head?”
Imagine that I’m a student studying under Feynman. I know that he’s one of the great minds of his generation, but it’s hard to tell which things I need to pick up. His internal thoughts are not very visible. In conversation with mathematicians, I see him easily catch errors in their claims, but I don’t know how he does it. I could just ask him how he does it, but he might not know; a young Richard Feynman probably just implicitly assumes that everyone pictures examples in their head, and has no idea why most people are unable to easily catch errors in the claims of mathematicians!
But if I ask him “what were you tracking in your head, while talking to those mathematicians?” then he’s immediately prompted to tell me about his hairy green ball thing.
More generally: for purposes of learning/teaching, the key question to ask of a mentor is “what are you tracking in your head?”; the key question for a mentor to ask of themselves is “what am I tracking in my head?”. These extra-information-tracking skills are illegible mainly because people don’t usually know to pay attention to them. They’re not externally-visible. But they’re not actually that hard to figure out, once you look for them. People do have quite a bit of introspective access into what extra information they’re tracking. We just have to ask.
Returns to Excess Cognitive Capacity
Mentally tracking extra information is exactly the sort of technique you’d expect to benefit a lot from excess cognitive capacity, i.e. high g-factor. Someone who can barely follow what’s going on already isn’t going to have the capacity to track a bunch of other stuff in parallel.
… which suggests that extra-information-tracking techniques are particularly useful investments for people with unusually high g. (Hint: this post is on LW, so “unusually high g” probably describes you!) They’re a way to get good returns out of excess cognitive capacity.
The same argument also suggests a reason that teaching methods aren’t already more focused on mentally tracking extra information: such techniques are probably more limited for the median person. On the other hand, if your goal is to train the great minds of the next generation, then figuring out the right places to invest excess cognitive capacity is likely to have high returns.
Other Examples?
Finally, the obvious question: what extra information do you mentally track, which is crucial to performing some task well? If the hypothesis is right, there’s probably high-value mental-tracking techniques which some, but not all, people reading this already use. Please share!