Episode 27: Problem Formulation
So we have been looking at the cognitive science of intelligence, and we've been looking at the seminal work of Newell and Simon, and we've seen how they are trying to create a plausible construct of intelligence to drive many different ideas together into this idea of 'intelligence as the capacity to be a general problem solve' and then they're doing a fantastic job of applying the naturalistic imperative which helps us avoid the homuncular fallacy, because we're trying to analyze, formalize, and mechanize our explanation of intelligence, explaining the mind ultimately in a non-circular fashion by explaining it in non-mental terms.
This will also hopefully give us a way of resituating the mind within the scientific worldview. We saw that at the core of their construct was the realization via the formalization and attempted mechanization of the combinatorial explosive nature of the problem space and how crucial relevance realization is and how somehow you zero in on the relevant information. They proposed a solution to this that has far-reaching implications for our understanding of meaning cultivation and of rationality; they propose the distinction between heuristic and algorithmic processing and the fact that most of our processing has to be heuristic in nature.
It can't pursue certainty, it can't be algorithmic, it can't be Cartesian in that fashion, and that also means that our cognition is susceptible to bias. The very processes that make us intelligently adaptive help us to ignore the combinatorial explosive amount of options and information available to us are the ones that also prejudice us and bias us so that we can become self-deceptively misled.
They deserve to be seminal figures and they exemplify how we should be trying to do cognitive science and the exemplify the power of the naturalistic imperative, but there were serious shortcomings in Newell and Simon's work. They themselves (and this is something we should remember, even as scientists: the scientific method is a psychotechnology designed to try to help us deal with our proclivities towards self-deception) fell prey to a cognitive heuristic that biased them. They were making use of the essentialist heuristic, which is crucial to our adaptive intelligence. It helps us find those classes that do share an essence and therefore allow us to make powerful predictions and generalizations.
Of course the problem with essentialism is precisely it is a heuristic, it is adaptive; we are tempted to overuse it and that will make us mis-see many categories do not possess an essence, like Wittgenstein famously pointed out the category of 'game' or 'chair' or 'table'. Newell and Simon thought that all problems were essentially the same, and because of that how you formulate a problem is a rather trivial matter. Because of that, they were blinded to the fact that all problems are not essentially the same, that there are essential differences between types of problems, and therefore problem formulation is actually very important.
This is the distinction between well-defined problems and ill-defined problems. I made the point that most real-world problems are ill-defined problems; what's missing in an ill-defined problem is precisely the relevance realization that you get through a good problem formulation. We then went into work in which Simon himself participated, the work of Kaplan and Simon to show that this self-same relevance realization through problem formulation is at work at addressing combinatorial explosion. We took a look at the problem of the mutilated chessboard, and that if you formulate it as a covering strategy you will get into a combinatorial explosive search whereas if you formulate it as a parity strategy, if you make salient the fact that the two removed pieces are the same color, then the solution becomes obvious to you and very simple to do. Problem formulation helps you avoid combinatorial explosion and helps you deal with ill-definedness and this process by which you move from a poor problem formulation to a good problem formulation is the process of insight.
This is why we have seen throughout that insight is so crucial to you being a rational cognitive agent and that means that in addition to logic being essential for our rationality those psychotechnologies that enhance our capacity for insight are also crucially important (indispensable!).
John Vervaeke has a lecture series on YouTube called Awakening from the Meaning Crisis. I thought it was great, so I'm arranging a lecture club to discuss it here on Less Wrong. The format is simple: each weekday I post a comment that's a link to the next lecture and the summary (which I plan on stealing from the recap at the beginning of the next lecture), and then sometimes comment beneath it with my own thoughts. If you're coming late (even years late!) feel free to join in, and go at whatever pace works for you.
(Who is John Vervaeke? He's a lecturer in cognitive science at the University of Toronto. I hadn't heard of him before the series, which came highly recommended to me.)
I split the lecture series into three parts: the philosophical, religious, and cultural history of humankind (25 episodes) related to meaning, the cognitive science of wisdom and meaning (20 episodes), and more recent philosophy related to the meaning crisis specifically (5 episodes). Each episode is about an hour at regular speed (but I think they're understandable at 2x speed). I am not yet aware of a good text version of the lectures; I also have some suspicion that some important content is not in the text itself, and so even if I transcribed them (or paid someone to) it'd still be worth watching or listening to it.
I think the subject matter is 1) very convergent with the sort of rationality people are interested in on LW, and 2) relevant to AI alignment, especially thinking about embedded agency.
Discussion: