Both the swampman and the spontaneous textbook are optimized, just by the inventor of the thought experiment rather than by anything that happens inside the thought-experument-world.
Suppose you encountered a swampman (in the sense of seeing a random person appearing out of nothing in a saltmarsh) but in this case we, the thought experimentors, draw randomly from the set of quantum fluctuations which are physiologically human and can speak English. The vast majority of "thoughts" and "memories" that a swampman would report would be utter nonsense, and not correspond to logic or reality.
Likewise, a randomly-fluctuated-into-existence-written-in-English-textbook would mostly contain false statements, since most possible statements are false.
If you did actually (by some insane miracle) encounter either of these things in real life, your priors should be on the random nonsense case, not the "it also happens to appear optimized even more" case.
(That is, if you were sure that the swampman/textbook was created by random fluctuations. If you actually see a person assembled from nothing in a swamp, you should probably freak out and start assigning high probabilities to God, aliens, simulators and---the big one---your own insanity.)
By positing a swampman with a coherent brain, or a textbook filled with true facts, there genuinely is an optimization pressure and it is you!
Causal explanations are a special case of acausal explanations, things like Solomonoff induction that posit hypotheses for some data/process/computation and don't care about the actual history of the thing. Acausal explanations enable coordination, multiple things of different natures and origins simulating shared computational (partial) reasons for their content. Actual history of a thing is merely one way in which it can be explained, it doesn't seem crucial to distinguish.
Acausal explanations can themselves be agents, their decisions jointly controlling all the things they serve as explanations for (are simulated or reasoned-about by). This way, we recover intentional stance, even for fixed data (via Solomonoff induction), and can ask how to influence coordination by incentivising the agentic acausal explanations for the things that we want to coordinate. This kind of decision relevant representation of a thing (in terms of the acausal explanations it admits) seems like the right way of establishing its meaning.
One way to patch/augment the distinction between original and derived intentionality is to allow for viewing phenomena with derived intentionality as doing some optimization/selection decohered from the purpose of the original intentional phenomenon (inner/mesa-optimizer-like).
For example, Process 1 with original intentionality ("intentionally") spawns Process 2 with derived intentionality, which ("non-intentionally") spawns Subprocess 2a, which then (again, "intentionally") spawns Process 3.
If we draw a phylogenetic tree messy, directed mycelium of optimization processes, some "children" will be "legitimate" ("intentionally spawned"), whereas other "children" will be "illegitimate" ("non-intentionally spawned").
It's not as much about giving something "special status", as rather about qualifying the type of parent-children relationship between two processes/artifacts. (It's probably also possible to fold the cases where there's no distinction between the optimizer and the optimizee.)
My own feeling is that this original/derived idea is somewhat chasing a ghost, like needing there to be something special which elevates the goals of a human above the purposes of a hammer. Granted, there are differences between how humans have goals and how a hammer has a purpose. Original vs derived just seems to me like the wrong frame for that.
I would like to hear you elaborate on that.
the swamp-man's thoughts are optimized by the thought-experimenter
This resembles the old p-zombies sequence.
I didn't read as carefully as I'd like, but some quick comments:
(I was going to just text you this but figured I should practice posting sometimes - catching some of your Inkhaven spirit maybe!)
Thanks for the thoughtful and detailed response!
I've sketched my own view elsewhere in this post as
Strictly speaking, teleosemantics is a theory of what constitutes semantic content, but I’m using it in this essay to talk about how states acquire semantic content. My model is that teleosemantic optimisation is how systems typically acquire semantic content but, once acquired, the constitution of mental content is played by the causal/functional roles of the concepts themselves.
This seems pretty close to your "structurally-optimised" picture. However, where we might diverge is that I'm aiming for a more realist notion of content whereas (if I'm reading you correctly) you're happier with a more deflationary view i.e. we can treat systems as if they have content using the Intentional Stance without there necessarily being a stance-independent fact of the matter about whether there's content.
Despite the similarity in our views, there's one bullet I'd try to avoid biting if possible:
The cost is affirming that "1+1=2" means something even if it is written by accident via the random movements of particles.
The original formulation of teleosemantics is meant to block exactly this. Roughly, content is located between a producer and consumer mechanism. For example, a frog has a visual system which produces states that lock onto flies and a motor system which consumes those states by snapping its tongue at them. In this case the content [fly] takes on meaning as it lives between producer and consumer mechanisms which participate in an optimisation process. In this way, you can say that the producer produces content for the consumer.
I think you can anchor this in the causal-functional roles played by the producer/consumer mechanisms rather than the fact that they've undergone a specific optimisation process i.e. we don't require the content to have actually been produced/consumed by an optimisation process but only that it plays the right causal-functional role.
In this way, a random configuration of space dust that happens to arrange into "1+1=2" doesn't have content in its own right because there is no sophisticated producer mechanism. Even if an agent comes along later to read the dust, the representational content is in the agent's existing structure and the dust pattern is just an "empty vehicle" that can be co-opted for the system's own existing representations rather than a genuine contentful state.
This contrasts with Swampman who instantiates a full producer/consumer architecture which is structurally and functionally isomorphic to our own. On my functional-role picture this is enough to constitute genuine content even though he never underwent any actual optimisation process that is normally needed to acquire the content.
To be clear, this view still probably has some issues to work through e.g. I'm probably drifting towards a type of semantic internalism which a good chunk of philosophers will want to reject. I'm also not sure this view fully evades underdetermination worries e.g. which consumer has the process been optimised for? In regular teleosemantics the consumer appears in the historical selection process but in this view there could, in principle, be many consumers which explain the same process.
Epistemic status: trying to articulate my own thoughts. I have not done a thorough literature review.
James Diacoumis commented on yesterday's post:
the typical objection to such versions of teleosemantics are swamp-man counterexamples: suppose a thermodynamic miracle occurred, with a perfectly formed human spontaneously assembling out of matter in a swamp. This person's thoughts cannot be ascribed semantics in a way that depends on evolution. My version of teleosemantics would be comfortable ascribing meaning to such a person's thoughts, because those thoughts would still be well-understood as being optimized for map-territory correspondence, much like a chess grandmaster's moves are well-explained by the desire to win.
Swampman’s thoughts haven’t been optimised for map-territory correspondence because Swampman hasn’t actually undergone the optimisation process themselves.
If the point is that it’s useful to describe Swampman’s thoughts using the Intentional Stance as if they’ve been optimised for map-territory correspondence then this is fair but you’ve drifted away from teleosemantics because the content is no longer constituted by the historical optimisation process that the system has undergone.
To recap some context:
Teleosemantics is a theory of mind which attributes meaning (semantics) to mental states based on biological purpose. My personal interest in teleosemantics de-emphasizes the biological aspect and focuses on the claim that the meaning of something is what it has been optimized to reflect.[1] Empirically, we can "ground out" all the purposes we see around us into biological evolution. An LLM, for example, was optimized by a gradient-descent algorithm; the gradient-descent algorithm was optimized by a human; the human was optimized by evolution. However, I don't see a need to stipulate this in the theory of semantics.
The Swampman example is a potential counterexample to teleosemantic theory. The example postulates that a man spontaneously forms in a swamp (a thermodynamic miracle due to quantum fluctuation or something along those lines -- vastly improbable but not strictly impossible, quite similar to a Boltzmann Brain). Since the Swampman's purposes cannot be grounded out in natural selection, this poses a potential challenge to versions of teleosemantics that consider this important: if you think Swampman's thoughts are as meaningful as an ordinary man's, this appears to be a counterexample to teleosemantics.
However, I went further in my statement yesterday. To quote it again:
My version of teleosemantics would be comfortable ascribing meaning to such a person's thoughts, because those thoughts would still be well-understood as being optimized for map-territory correspondence, much like a chess grandmaster's moves are well-explained by the desire to win.
James Diacoumis is pointing out a potential problem with the way I'm interpreting "optimization" in the teleosemantic "optimized to reflect". The phrase "have been optimized" can have a connotation of "have been optimized by ___" (we can attribute the optimization to some optimization process, such as a person, or natural selection). In contrast, the sense I had in mind when writing was one which doesn't require an optimizer to be identified: rather, the optimization is identified as a predictively good hypothesis (we can predict many features of Swampman by predicting that they'll score highly on ordinarily biological purposes).
Let's call "have been optimized by ___" style optimization "historically-optimized" (there's an optimizer in its history) and the alternative "structurally-optimized" (the structure of the thing reflects optimization).
I can see a reasonable case for both alternatives, to the point where I currently see it as a take-your-pick situation.
The advantage of historical-optimization, as I see it, is that it won't count coincidental configurations of dust, gasses, etc as "meaningful". Atoms of gas go through many many configurations as they rapidly bounce around. Moreover, there's a whole lot of gasses spread through the universe. Somewhere, at some point in time, some configurations of these will yield meaningful-looking configurations with high map-territory fit, such as "1+1=2". Historical-optimization allows you to rule these cases out, while structural-optimization does not.
You might think that a coincidental "1+1=2" won't "look optimized" in the structural-optimization sense because the hypothesis isn't predictive: on the whole, the usual hypothesis about gasses being randomly configured will do well, and a hypothesis which expects meaningful messages will do poorly.
I think such a defense of structural-optimization probably won't work, because we need to be able to look at things individually.[2] The purposeful hypothesis won't do well at predicting gas-in-general, but it'll do great for this isolated patch of gas at the specific moment in time where it forms the symbols "1+1=2".
Obviously, this is very similar to the Swampman. Swampman gets their improbability by fiat, as a philosophical thought-experiment. The "1+1=2" written in gas particles gets its improbability from searching through the vastness of space and time. However, they're both just low-probability events which create structural-optimization without historical-optimization. If you want such cases to count as having semantics, it seems natural to choose structural-optimization. If you want such cases to lack semantics, it seems natural to choose historical-optimization.
If you look at it that way, it seems like not so much a dilemma as a choice. (James Diacoumis insists that choosing structural-optimization discards teleosemantics, which I think is probably historically accurate; however, it doesn't discard the insight I care about.)
If I were to choose historical-optimization, I would be mostly OK with accepting that Swampman doesn't have thoughts which refer by this definition, because the thought experiment depends on a thermodynamic miracle and this doesn't seem important for AI safety. This is similar to several responses to Swampman that appear in the literature: in The Status of Teleosemantics, or How to Stop Worrying about Swampman David Papineau argues that teleosemantics is a naturalistic theory, similar to "water = H2O", and only has to account for empirical reality, not every hypothetical we can come up with (comparing the Swampman argument to hypothesizing a scenario where a different chemical behaves entirely like water -- the mere hypothetical doesn't start to be an argument against "water = H2O").
If one insists on historical-optimization, but wishes for a theory maintaining that the Swampman's thoughts can in fact refer, then I am tempted to suggest that the historical-optimization comes from outside thought experiment itself; IE, the swamp-man's thoughts are optimized by the thought-experimenter. It isn't entirely clear to me that this suggestion doesn't work. It is similar to suggesting, in the case of the "1+1=2" spelled in gasses, that the optimization comes from the search across space and time for the special occurrence.
More seriously, though: maybe the Swampman's thoughts are initially non-semantic when they first spontaneously appear, but they quickly get optimized (by the brain) for better map-territory correspondence as they look around, think about their situation, etc. In other words, the brain itself is the optimizer which optimizes the thoughts. This is unlike the "1+1=2" example, which falls apart immediately. This route is available to me, though not to the more typical teleosemantic theories which ground out in evolution alone, even if I restrict myself to historical-optimization.
It seems that those interested in teleosemantics are often quite concerned with a distinction between original intentionality vs derived intentionality. For example, A Mark of the Mental: In Defense of Informational Teleosemantics discusses this distinction in chapter 1.
For example, a hammer has derived intentionality: it was made for the purpose of hitting nails by thinking beings who gave it that purpose as a means to their ends.
It does seem natural for teleosemantic theories to grapple with this question, particularly if they choose historical-optimization. After all, teleosemantics isn't typically just about semantics; someone inclined towards teleosemantics will typically choose a similar theory for all purposes (grounding purpose in having-been-optimized-for). Yet, this would appear to make all intentionality derived intentionality. To avoid an infinite regress, we would appear to need to come up with some source of original intentionality.
Based on my limited understanding, it would seem natural for teleosemantic theories to ascribe original intentionality to natural selection (and only natural selection). However, this does not seem to be the case. A Mark of the Mental ascribes original intentionality to humans as well. This seems a bit baffling, for a theory which grounds out purposes in natural selection. Clearly I need to read more about these theories. (I've only read a little bit of A Mark of the Mental.)
My own feeling is that this original/derived idea is somewhat chasing a ghost, like needing there to be something special which elevates the goals of a human above the purposes of a hammer. Granted, there are differences between how humans have goals and how a hammer has a purpose. Original vs derived just seems to me like the wrong frame for that.
The benefit of choosing structural-optimization, so it seems to me, is that one does not need to worry about any infinite regress. Some optimization-like processes can arise by chance, and some of those can select for yet-more-optimization-like processes, and purpose can arise by increments. We don't especially need to worry about original purpose vs derived purpose; it needn't play a significant role in our theory. Those who prefer to affirm that Swampman refers don't need to think about any of my arguments which tried to defend historical-optimization; structural-optimization naturally identifies Swampman as optimizing map-territory correspondence.
The cost is affirming that "1+1=2" means something even if it is written by accident via the random movements of particles.
The truth is, while I do think the above arguments make some good points, I think the discussion here was too loose to draw any firm conclusions. There are far more notions of "optimization" to consider than just two, and it seems plausible that some of my conclusions would be reversed upon more careful consideration. Furthermore, I haven't surveyed the existing literature on teleosemantics or Swampman-arguments adequately.
What about a lie? If I lie to you and say "the sun is green today" then I haven't optimized this statement for map-territory correspondence. Specifically, I haven't optimized my statement to correspond to a territory in which the sun is green today. Nonetheless, this is clearly the meaning of my statement. Is this a counterexample to teleosemantic theories?
On my current account, we can find the semantics by looking at the society-wide optimization process which attempts to keep words meaningful. This includes norms against lying, as well as norms about the mis-use of words. If I say "wire" to mean hose, someone will correct me. Someone could correctly say "Abram means hose" (there is an optimization process which is optimizing "wire" to correspond to hoses, namely myself), but it would also be correct to say "'wire' does not mean hose" (there is another, more powerful optimization process working against the map-territory correspondence I'm trying to set up).
A different way of trying to deal with lies would be to look at what something is optimized to communicate. If I were trying to convince someone of "the sun is green today" you might try to ground out my meaning in what I'm trying to do to that person's mind. However, if we take this as our general theory of semantics, then notice that we have to analyze the meaning of the thoughts in the listener's mind in the same way: thoughts have meaning based on what they try to communicate (to the future-self, or perhaps to others). Meaning would be founded in this infinite chain into the future. I'm not inherently against circular definitions (we can view this as a constraint on meaning), but this doesn't seem like a great situation. (The constraints might end up not constraining very much.)
I'm actually not too sure about this. Maybe we need to consider our understanding of an object in context. This seems like it might be resolved at the level of "what are we trying to make the theory for" rather than something we can resolve in the abstract.