This post was educational, however, I want to push back against the implicit criticism of instrumentalism and the Copenhagen interpretation. The metaphilosophical position I will use here is: to solve a philosophical question, we need to rephrase it as a question about AI design (AFAIK the full credit for this approach goes to Yudkowsky). So, suppose I am building an AI. Should the AI's model of the world be (i) "about things that exist independently of the AI" or (ii) "about the subjective perceptions of the AI"? This depends on what kind of reward function I want my AI to have.

The standard (I call it "perceptible") type of reward function in reinforcement learning only depends on the history of actions and observations. For such an AI the answer is (ii): it is important the AI will correctly predict the consequences of its actions, but there is no significance whatsoever that the AI's models can be interpreted as (i). Yes, these models will still have intrinsic variables corresponding to "unobserved objects" in some sense, but there is no reason for these variables to always have an unambiguous "realist" interpretation.

Now suppose that the AI is actually designed to care about particular objects outside itself. Specifically, assume the AI uses an instrumental (or semi-instrumental) reward function. Such a function might be specified (or partially specified) using some particular ontology. Then, the AI is modeling the world as containing certain unobserved objects, at least approximately. The answer is now in between (i) and (ii). The AI's models are about subjective perceptions of the AIs and also about some particular type of things that exist independently of the AI, namely, those things that are normatively important to it. Coming back from AIs to humans, we may conclude that, what really makes sense is our subjective perceptions + those external objects that we actually care about (e.g. other humans). But, quarks only "exist" in the sense that they are components in a useful model we created.

What does it tell us about the interpretation of quantum mechanics? Once again, consider an AI trying to discover quantum mechanics. From the AI's perspective, what it's looking for is a function from observation histories to distributions over the next observation. How can we construct such a function from the formalism of QM? Obviously, using Copenhagen: each observation is a measurement that causes the wavefunction of the environment to collapse. "But," the fans of MWI will object "what about all those other Everett branches? They are still out there, right? They don't actually vanish?!" The point is, from the AI's perspective the question is meaningless. The AI's design assumes the AI can record everything it observes in its memories, therefore, once an observation is made, those Everett branches will never meet again in the AI's lifetime. "But," another objection may go "what if someone tempers with the memories of the AI in a way that allows quantum interference between the branches? Sure it is completely impractical for humans, but it is theoretically possible, and might even be practically possible for an AI running on a quantum computer." Alright, but tempering with the memory of the agent is effectively destroying the agent: it invalidates the fundamental assumptions of its reasoning algorithm, and any reasoning algorithm must make some fundamental assumptions of that sort. (The agent might still accurately account for the possibility of its own destruction (see "The Death of the Agent and Kamikaze Strategies"), but probably only with the help of external knowledge.)

Showing 3 of 5 replies (Click to show all)
2Rob Bensinger8moWhat does it mean for instrumentalism to be the correct metaphysics? Normally, I'd interpret "the correct metaphysics" as saying something basic about reality or the universe. (Or, if you're an instrumentalist and you say "X is the correct metaphysics", I'd assume you were saying "it's useful to have a model that treats X as a basic fact about reality or the universe", which also doesn't make sense to me if X is "instrumentalism".) Well, sufficiently specific realist desiderata. Adding hidden variables to QM doesn't make the theory any more realist, the way we're using "realist" here.
2Vanessa Kosoy8moLike I said before, it means that instrumentalism is the point of view that is the most useful for designing AI or answering questions about AI. According to the "Yudkowskian computationalist" metaphilosophical view, this also makes it the most useful for rationality in general. I imagined "realist" to mean something like "the universe can be described in a way independent of the choice of observer, and the perceptions of any given observer can be decoded from the history of the universe in this description, s.t. different observers have compatible observations". Adding hidden variables does make QM more realist in this sense, for example the de Broglie-Bohm interpretation is realist (but it only makes sense if you assume all observer perceptions can be reduced to configuration variables, which seems false and disqualifies it). MWI fails to be entirely realist because you have to either make the decoding of observer perceptions stochastic (and thereby admit that your description of the universe is incomplete) or to postulate, for each "normal" observer Alice, a huge ensemble of different observers that correspond to versions of Alice in different Everett branches (and thereby lose the physical meaning of quantum probabilities and with it essentially all the predictive power of the theory).

Like I said before, it means that instrumentalism is the point of view that is the most useful for designing AI or answering questions about AI. According to the “Yudkowskian computationalist” metaphilosophical view, this also makes it the most useful for rationality in general.

Except that if you are the kind of rationalist who cares about what is really real, you should reject instrumentalism immediately.

A simple sketch of how realism became unpopular

by Rob Bensinger 3 min read11th Oct 201953 comments


[Epistemic status: Sharing current impressions in a quick, simplified way in case others have details to add or have a more illuminating account. Medium-confidence that this is one of the most important parts of the story.]

Here's my current sense of how we ended up in this weird world where:

  • I still intermittently run into people who claim that there's no such thing as reality or truth;
  • a lot of 20th-century psychologists made a habit of saying things like 'minds don't exist, only behaviors';
  • a lot of 20th-century physicists made a habit of saying things like 'quarks don't exist, only minds';
  • there's a big academic split between continental thinkers saying (or being rounded off to saying) some variant of "everything is culture / perception / discourse / power" and Anglophone thinkers saying (or being rounded off to saying) "no".

Background context:

1. The ancient Greeks wrote down a whole lot of arguments. In many cases, we're missing enough textual fragments or context that we don't really know why they were arguing — what exact propositions were in dispute, or what the stakes were.

2. In any case, most of this is screened off by the fact that Europe's memetic winners were Christianity plus normal unphilosophical beliefs like "the sky is, in fact, blue".

3. Then, in 1521, the Protestant Reformation began.

4. In 1562, the Catholics found a giant list of arguments against everything by the minor Greek skeptic Sextus Empiricus, got very excited, and immediately weaponized them to show that the Protestant arguments fail (because all arguments fail).

5. These soon spread and became a sensation, and not just for being a useful superweapon. Plenty of intellectuals were earnest humanists used to taking arguments at face value, and found Sextus' arguments genuinely upsetting and fascinating.

I trace continental thinkers' "everything is subjective/relative" arguments back to a single 1710 error in George Berkeley:

[...] I am content to put the whole upon this Issue; if you can but conceive it possible for one extended moveable Substance, or in general, for any one Idea or any thing like an Idea, to exist otherwise than in a Mind perceiving it, I shall readily give up the Cause[....]
But say you, surely there is nothing easier than to imagine Trees, for instance, in a Park, or Books existing in a Closet, and no Body by to perceive them. I answer, you may so, there is no difficulty in it: But what is all this, I beseech you, more than framing in your Mind certain Ideas which you call Books and Trees, and the same time omitting to frame the Idea of any one that may perceive them? But do not you your self perceive or think of them all the while? This therefore is nothing to the purpose: It only shews you have the Power of imagining or forming Ideas in your Mind; but it doth not shew that you can conceive it possible, the Objects of your Thought may exist without the Mind: To make out this, it is necessary that you conceive them existing unconceived or unthought of, which is a manifest Repugnancy.

If I can imagine a tree that exists outside of any mind, then I can imagine a tree that is not being imagined. But "an imagined X that is not being imagined" is a contradiction. Therefore everything I can imagine or conceive of must be a mental object.

Berkeley ran with this argument to claim that there could be no unexperienced objects, therefore everything must exist in some mind — if nothing else, the mind of God.

The error here is mixing up what falls inside vs. outside of quotation marks. "I'm conceiving of a not-conceivable object" is a formal contradiction, but "I'm conceiving of the concept 'a not-conceivable object'" isn't, and human brains and natural language make it easy to mix up levels like those.

(I can immediately think of another major milestone in the history of European thought, Anselm's ontological argument for God, that shows the same brain bug.)

Berkeley's view was able to find fertile soil in an environment rife with non-naturalism, skeptical arguments, and competition between epistemic criteria and authorities. Via Kant and Kant's successors (Hegel chief among them), he successfully convinced the main current of 19th-century European philosophy to treat the idea of a "mind-independent world" as something ineffable or mysterious, and to treat experiences or perspectives as fundamental.

(Edit: G.E. Moore seems to think that everyone in the 19th century was making an error along these lines, but I now suspect Kant himself wasn't making this mistake; I think his main error was trying too hard to defeat skepticism.

I also don't think Berkeley's writing would have been sufficient to confuse Europe on its own; it's too lucid and well-articulated. The transition to Kantian and Hegelian versions of these arguments is important because they were much more elaborate and poorly expressed, requiring a lot of intellectual effort in order to spot the inconsistencies.)

My unscholarly surface impression of the turn of the 20th century is that these memes ("the territory is fundamentally mysterious" and "maps are sort of magical and cosmically important") allowed a lot of mysticism and weird metaphysics to creep into intellectual life, but that ideas like those are actually hard to justify in dry academic prose, such that the more memetically fit descendants of idealism in the 20th century ended up being quietist ("let's just run experiments and not talk about all this weird 'world' stuff") or instrumentalist / phenomenalist / skeptic / relativist ("you can't know 'world' stuff, so let's retreat to just discussing impressions; and maybe you can't even know those, so really what's left is power struggles").

Today, the pendulum has long since swung back again in most areas of intellectual life, perhaps because we've more solidly settled around our new central authority (science) and the threats to centralized epistemic authority (religious and philosophical controversy) are more distant memories. Metaphysics and weird arguments are fashionable again in analytic philosophy; behaviorism is long-dead in psychology; and quietism, non-realism, and non-naturalism at least no longer dominate the discussion in QM, though a lot of Copenhagen slogans remain popular.

The above is a very simple picture featuring uneven scholarship, and history tends to be messier than all that. (Ideas get independently rediscovered, movements go one step forward only to retreat two steps back, etc.) Also, I'm not claiming that everyone endorsed the master argument as stated, just that the master argument happened to shift intellectual fashions in this direction in a durable way.