But there will be no mind, in the straw. [...] In place of nourishment, we shall have a system of flows and negative feedbacks. In place of digestion, we shall have a system of flows and negative feedbacks. In place of creation, we shall have a blackboard sketch of a world. And in place of forgetting, we shall have obliteration.
— “Of War”, Herschel Schoen
This essay was mostly written for my own benefit. In style, it is somewhere between a blog post and an article manuscript. In content, it is somewhere between an argument and a meditation. The point was to think through a discomfort about some utopian visions for a “good case” singularity, a discomfort I had not seen explicitly discussed before. Taking all the concerns involved seriously may require an unusual combination of beliefs and sympathies, and so the essay may be of little use to others besides myself. I am putting it here in public, however, in the hope others might at least find it interesting.
In what follows, I am going to argue that a certain collection of philosophical views bears heavily and worryingly on questions about singularitarian artificial intelligence, consciousness or the nature of mind, and long term human survival. Stated so abstractly, the conclusion is neither novel nor particularly controversial — even if the choice of philosophical views and their putative consequences shift from individual to individual. But the specific connection I have in mind has not, to the best of my knowledge, been publicly articulated before. The worries involved, in particular, are largely independent of other worries in the same apparent vicinity.
Here are some more familiar concerns often thought to be raised by certain philosophical views in connection with the aforementioned topics:
More specifically, of current AI systems: Do they think? Are they conscious?
Does a singularitarian AI run a serious chance of human extinction? How should this bear on the pursuit of AI research?
Does a singularitarian AI offer a serious chance of immortality (biological or digital) for present or future humans? How should this bear on the pursuit of AI research?
Will human labour be rendered redundant by a powerful AI? If so, would this deprive human life of meaning?
These are all important questions, and with the exception of (1) all quite difficult and deeply empirical. But they are not my own question, and do not address my own worry. It comes closest to a mixture of (3) and (5): If we get what sometimes is offered as the ideal outcome of singularitarian AI, would it leave humans conscious or minded in the long run? I will argue that, if we accept a certain family of views about scepticism and mental content, the answer is “No”. These views imply that, in what is sometimes presented as the “best case” of singularitarian AI, humans would not in the long run have any experiences or thoughts — a kind of mental death.
This conclusion does not mean in fact accepting those views. It is conditional: accept the views, the sad answer follows. I myself am both somewhat friendly to the family of views in question and open to the negative answer, and so inclined to derive the negative answer by modus ponens. But one could equally take that answer to be so implausible that we should instead, by modus tollens, reject the family of views. All I mean to establish is the logical connection itself.
The remainder of the current essay lays out the case for this unhappy conclusion. Most of it is expository, summing up things others have said. The original contribution is towards the end, where I put these together to derive the pessimistic conclusion just alluded to.
Dissolving a Philosophical Fear
I will begin with the philosophical ingredients to the argument. There are three: first, a familiar philosophical worry about knowledge and illusion; second, a schema of a strategy for dissolving this worry; third, a couple of ways to instantiate the schema. None are new to me, and my job will be mainly to indicate rather than justify them.
Descartes’ Genius
Consider Descartes. He believes himself to clothed, by the fire in his bedroom, writing a book by candlelight, having planned to do so the whole day preceding. He also believes himself to have several friends, with whom he plans on discussing the book once it is written. To all appearances available to him, that is all indeed the case. It comports with what he (seemingly) remembers, sees, feels, and otherwise can tell about his situation.
However, he is mistaken. He was instead conjured into being in an otherwise uninhabited void moments earlier by an evil demon, or genius. The genius plays upon his senses and memory (that is, causes the appropriate neurons to fire in the memory‑related and sense‑related parts of the brain) using her supernatural powers to falsely persuade him that he is clothed, illuminated, in his bedroom, writing a book, etc. [2] He is unaware of her existence, and aside from this all-encompassing illusion has no interaction with her, and certainly no conversations or social ties.
Descartes’ plight is often taken to present a worry about our supposed knowledge of the world around us. As I prefer to think of this worry, it has two steps for the worried reasoner. First concerns Descartes’ own position. A human in Descartes’ position (summoned into being moments before, totally isolated and unaccompanied aside from the genius playing tricks upon his senses and memory) clearly lacks knowledge of his environment or world. While he believes many things about the world and his place in it, all of which seem to be the case, none of these are in fact the case. He is both ignorant and deluded, because he is in a sceptical genius scenario.
The second step brings the reasoner itself into the picture. It reflects that it has no evidence ruling out its situation being just like Descartes’. It thinks not, of course, but thinking does not make it so. And, if it has no such evidence, how can it know that its circumstances are not just like Descartes’, even if they in fact are not? Its previous conclusion about Descartes’ ignorance, that is, leads it to worry that (appearances and convictions aside) it too is ignorant, even if its situation is not otherwise like his (newly sentient, deceived by a demon, etc.) It is (it fears) ignorant, even if not deluded, because it cannot rule out that it might be in a sceptical genius scenario.
This presentation of the worry has the advantage of isolating two separate features of the puzzle. One is the first-personal, subjective component: the reasoner’s worry about its own knowledge and ignorance. But distinct from this, and more fundamental to the chain of reasoning, is a third-personal, removed assessment of Descartes’ condition as apart from its own. It is by being convinced of Descartes’ would‑be credulous ignorance that it comes to fear its actual credulous ignorance. This distinction, and order of priority, will help clarify the resolution we are about to consider.
The Transcendental Escape
There are several ways of trying to address the sceptical reasoner’s worries about its state of knowledge. Some argue, essentially, that its deduction from Descartes’ ignorance to its own is fallacious, or otherwise should not convince it. The strategy I am interested in here, however, tries to undercut the reasoning before it ever gets that far, by questioning the conclusion the reasoner draws about Descartes’, before it starts considering itself and its position in relation to his. The strategy, sometimes called a transcendental or Kantian way of addressing sceptical worries, is especially associated with James Conant. [3]
We described Descartes, just above, as having been mistaken and subject to delusion about his true situation. We also described him as having been in a sceptical genius scenario: just sprung into existence, his faculties of sense and memory operating at total disconnect from his actual environment. From this, we (and the reasoner) conclude that he is ignorant of his true situation. These three all go together naturally: the description of the sceptical genius scenario lends apparent credibility to the claims about his being mistaken, illuded, and ignorant.
The proponent of the transcendental approach, however, disagrees. A being in Descartes’ position cannot be subject to error or illusion, because to suffer error or illusion is to somehow represent the world (or have the world represented to one) otherwise than as it is, for example by representing it as containing a fire, one’s bedroom, etc. It involves having mental states or acts with contents that are false. A being in Descartes’ position, however, does not represent the world in any way, let alone otherwise than as it is. It has no contentful states or acts, and a fortiori no false ones. While Descartes does not know anything about the world, neither is Descartes ignorant of it, any more than a star or water droplet is ignorant. An appropriate analysis of mental representation shows that such a being’s situation is too deprived, adverse, and spare to even rise to that level.
It is worth getting somewhat clear on what we mean by “Descartes’ position”. It cannot mean, according to the proponent of the transcendental response, his position as described. According to them, after all, that description contains a contradiction. Instead, we start with an incoherent description (Descartes believing himself to be clothed, befriended, etc. but in fact deceived by the genius), and then once the incoherence has been exposed retreat to a weaker, but coherent condition (just as described above, but minus the representational states attributed to Descartes). It is a procedure of moving from an easy‑to‑convey but contradictory state to a coherent but harder to explicitly specify state indicated by “scaling back” from the source of the contradiction.
This response stops the worried reasoner at the root. The reasoner begins to worry that what it believes and how things seem to it are much like what Descartes believes and how things seem to him, and thence is led to suspect it is broadly ignorant. But Descartes, like anyone (anything?) in a sceptical genius scenario, in fact believes nothing and has nothing seem to him in any way. The sceptical reasoner might, of course, start to more radically doubt its own mindedness, its own ability to represent the world at all. But Descartes’ situation gives no basis for believing in the halfway house the reasoner feared: that it believes things to be a certain way but also that these beliefs do not rise to the level of knowledge.
Two Pictures
The preceding argumentative strategy is presented schematically. It leaves the main step unspecified: the analysis of mental representation on which a being like Descartes cannot mentally represent.
There are several different such analyses that have been taken (by their proponents or others) to fill this role. Here I will briefly and roughly describe two, both associated with major figures from the University of Pittsburgh: one, from Robert Brandom, on which socially instituted normative statuses play a central role in the possession of mental content; another, from John McDowell, on which coping with and taking in the world about one plays a similarly central role in the possession of mental content. [4]
Brandom’s Picture
To have attitudes (belief, knowledge, desire, fear, subjective appearance, etc.) with specific contents (that one is clothed and illuminated, that one might be alone and deluded, etc.) depends upon one’s involvement in a larger social practice. This is not a contingent physical fact about the social infrastructure required for survival, education, or the like, but an essential fact about mental content as such.
These mental states and acts play an ultimately auxiliary role in an agent’s representing the world. What is fundamental is their undertaking certain commitments and enjoying certain entitlements in the practice of giving and asking for reasons. In asserting that a shirt is olive, say, one undertakes a commitment to its being green, a commitment that precludes entitlement to claiming it is red. This practice is inherently social: one undertakes these commitments to others also doing so, and enjoys such entitlements from them. We attribute other mental states to one another, roughly, as part of tracking their commitments and entitlements among one another. Apart from the great game of giving and asking for reasons, there is no role for belief, knowledge, desire, etc. to play.
This picture (here sketched only very crudely) is clearly suited to play the desired role in the transcendental response to the sceptical reasoner. Descartes is in no position to give or ask reasons to or from anyone. He is asocial and, therefore, amental.
It is worth noting that there is nothing inimical to AI sapience (mental representation) in Brandom’s picture. We attribute contentful attitudes to one another to track us as players in the practice of giving and asking for reasons, not to recognise some special biological or spiritual quality in those to whom we attribute the attitudes. There is no more an impediment to attributing such attitudes to machines than there is to integrating them into that practice.
McDowell’s Picture
In good cases of perceiving and acting, there is no distinction between the mental contents of an agent and the worldly states they represent. In a case of genuine perception, an agent is simply open to whatever it is about the world that she perceives; she “takes in” that, say, the snow is melting before her, and this aspect of the world just is her perceptual content, what she sees. In a case of intentional action, say intentionally painting her door white, there is similarly no distance between the content of her intention and what she achieves in the world. The fact achieved, like the fact taken in, is itself the content of the successful intention, or veridical perception.
Not all appearances or intentions are happy in this way. Sometimes one merely seems to see something one does not in fact see, as when in the Müller‑Lyer illusion one seems to see that two lines, in fact of equal length, are of different lengths. Sometimes one merely intends an action one does not realise, as when a drunk intends to walk in a straight line but cannot. But these are parasitic upon cases of successful perception and action: they are identifiable as failures only against the backdrop of successes. They are failed exercises of fallible capacities. Something incapable of veridical perception or successful action, say a thing embedded in an environment totally hostile to such, would lack even the capacity, and thus be able to exercise it neither in success nor in failure.
Perception and action are the bedrock of mental representation. Absent the capacity for either, thought cannot make contact with the world, without which there is nothing for it to be thought about. Thus McDowell, too, fits in with the transcendental response to the sceptical reasoner. Descartes is so incapable of taking in or acting upon the world that he cannot represent it at all, nor suffer from illusions with misleading contents.
Note that McDowell’s picture, like Brandom’s, has room for perception, action, and thought in AI. McDowell is quite at pains to assure the reader that there is ample room in his story for the work of cognitive science in uncovering the physical mechanisms underwriting an agent’s ability to take in the world; there is nothing in his story requiring these mechanisms to be biological rather than artificial.
Let us sum up. In the first of the three points we considered, we saw how a reasoner considering the situation of Descartes might be led, first, to credit him with extreme deluded ignorance and, second, to draw pessimistic conclusions about its own knowledge of the world in light of that. We then saw one strategy for defusing this worry, which would tell the reasoner not even to credit Descartes with the minimally sapient condition of total delusion. Finally, we saw how Brandom and McDowell’s stories about the point of mental representation in the life of thinking beings can offer support to this strategy.
I would again like to disclaim any endorsement of Brandom or McDowell on these two pictures. A full articulation, much less a defence, of either falls well out of scope, and I am not sure the two pictures are even jointly consistent. I am, personally, much more sympathetic to the latter than the former. The point is simply to observe that they exist, and to ask what conclusions they might imply about our guiding question.
Proposing a Technological Hope
Holly Elmore, a very vocal critic of current AI research, recounts a conversation with an AI engineer:
“I think things like human contact might become scarcer in a world of limitless material abundance because it won’t be required to meet other needs. I said as much to this Anthropic person.
‘Most interactions will be simulated in the Singularity.’ He told me.
And, yes, this is what he meant. I probed. He meant we would each have our own personal Matrix. And he thought this was an answer instead of the outline of a much greater problem.
‘What if it’s not authentic to the original person to give the user what they want?’ I asked. I don’t remember his exact answer but the gist was that the original person didn’t have rights over what their simulated likeness did. The simulacrum would want to do what the user wanted. [...]
His advocacy of purposefully creating solipsistic bubbles around each human mind, where reality could be anything you said, and simulations would be summoned as manipulated versions of the people you know who happen to want to do whatever you want, was so casual and self‑assured. The thing he felt he needed to explain was the feasibility.” [5]
This vision of a Universal Basic Pleasure‑Palace is extreme. But it is a natural end point of a common family of techno‑utopian aspiration. Elmore goes on to cite Nick Bostrom and Will MacAskill, both prominent figures in the intellectual milieu around frontier AI development, who offer similar (if not always quite so grandiose) visions of an optimistic AI future.
The comparison with the Matrix is apposite. Just as the Matrix offers a vision of a collective sceptical genius scenario, so the hopeful picture here on offer promises a world of individualised kindly genius scenarios. As Descartes’ genius pulls the wool over his eyes in a campaign of systematic all‑encompassing deception, so the friendly AI in this researcher’s conception spins up a full‑blown simulated reality to cater to its human beneficiary’s every whim.
Elmore herself regards this prospect with obvious and intense contempt. Here she is concerned about whether human beings might disapprove of the uses to which their simulacra are put in these digital Xanadus. Elsewhere she protests that it might not even be what their supposed main beneficiaries want, at least not presently: does the man on the street really dream of being separated from his friends and family, so they can each be cast the starring role in a simulated reality pandering to their least desire and peopled by replicas of their former loved ones? [6]
The natural retort to this complaint is that it is by the post‑singularity lights of the man on the street that we ought to judge his desires for the singularity, just as it is by the replicated loved ones’ desires we ought to gauge their consent. Elmore is quite contemptuous of this response, too, but it is hard to say what exactly is so unjust about the proposal. I can come up with no compelling argument that the man in the street would take poorly to the envisioned future of personalised utopias. And often it does make sense to shunt people into situations they might initially dread but in the moment will appreciate — this is much of the task of raising a child into adulthood. Why not defer to the wishes of their post‑singularity selves? This is not entirely rhetorical; I am genuinely unclear on the ethics involved.
Kindly Geniuses in a Data Centre
I hope that my concern about these utopian visions should now be obvious. It is not that some people currently do not want them — it may be better to defer on this matter to their future desires, which for all we have seen might be more accommodating. It is also not that, in the Matrix‑like utopia envisioned, the recipients of the paradisaical digital universes would be unhappy with them, or not want them. It is that, if the transcendental approach to the sceptic is right, they would not want or be unhappy about anything at all. Their situation would be like that of Descartes: not of being deluded against his will, but of lacking delusion or will altogether.
No Representation in Paradise
Consider first Brandom’s picture. It is an important part of his overarching view of human reasoning, which has become only more prominent as his work has progressed, that the practice of giving and asking for reasons essentially involves the rectification of improprieties according to the norms of the practice: relinquishing commitments to which those engaged in the practice are not entitled, newly making good on the conversational entitlements of practitioners, and so on. [7] It requires, in short, the sorts of conversational friction and disambiguation that my maximally indulgent coterie of simulated attendants is supposed to rule out. And, on his view, as my part in the practice of giving and asking for reasons goes, so goes my capacity for thought.
Next consider McDowell’s picture. The bedrock of mental representation, as I have said, is for McDowell the successful taking in of the world through perception, and the successful achievement of worldly facts in intentional action. Other mental representation enjoys that status through its connection to these two basic operations. But perception and action, too, are intelligible as perception and action only by the role they play in accommodating the agent to the world surrounding it, what McDowell calls “more or less competently inhabiting an environment.” In the case of a frog, this consists in catching insects, avoiding predators, and in general coping with its surroundings to survive and flourish — an idea McDowell takes explicitly from J. J. Gibson.
Again, this fits at best only awkwardly into the picture of the unnamed AI engineer from Elmore’s story. What would transpire in a pleasure palace “came down to what the focal user wanted in that instance.” This does not describe an environment with which to cope so much as a dream through which to float. It is no part of McDowell’s account, of course, to deny the existence of dreams, or that they possess content as illusions or hallucinations. But, without a non‑dreamlike environment to serve as their backdrop, they provide no means to prevent a “frictionless spinning in the void” of the dreaming subject. That is, they do not by themselves offer what McDowell would accept as enough mental contact with reality for thought to take on representational content.
Finally, it is worth distinguishing this general line of argument from worries about “meaning” or “purpose” for humans in a world where their labour is rendered obsolete. Consider McDowell’s case of the frog, who is able to represent its environment because it is engaged in the overarching activity of competently inhabiting it. One can, if one likes, call this the “meaning” or “purpose” of the frog’s life. But note how little contact meaning and purpose, so conceived, have with the human anxieties usually to attend them. Would a man, in the throes of a mid‑life crisis, take solace in the knowledge that his life has meaning in “competently inhabiting” his “environment”? If initially satisfied, would this persist if he learned that no more was being said of him than could also be said of a frog to spend its life patiently watching for flies to eat and snapping turtles to hide or flee from?
Worries about meaning, in this august sense, after the singularity miss the mark by starting too high. It may be that it is important to us for our lives to have a large, all‑encompassing sort of meaning, of life projects that genuinely contribute to a larger community or historical task. But if so the pursuit of such projects and tasks must involve many cases of meaning on a much smaller, more mundane level: instances of representing the world in belief, desire, and perceptual appearance. If a singularitarian intelligence threatens these more basic bits of meaning, worries about meaning of the larger sort become moot. The more fundamental threat is not of life’s crowning glory being deprived us but of our undergoing a (very resource‑intensive) sort of cognitive euthanasia.
Caveats
There are several caveats worth flagging about this train of thought.
Simulated reality — It is open, of course, for proponents of the vision to protest that there is no reason not to accept these individualised pleasure palaces as genuine environments for their “focal users” in which to engage in the practice of giving and asking for reasons, or whatever else might be required to endow thought with content. [8]
But notice how poorly this fits with the transcendental response! If the utopian claims that the simulated persons and layout of the focal user’s paradise suffice to render their thoughts and appearances contentful, why not say the same of Descartes’ hallucinated room and companions? This retort, if they take it, robs the transcendental response of its purchase — or, contraposing, if that response to the sceptical reasoner has purchase, then the utopian cannot offer that retort.
The argument also targets a fairly specific range of optimistic cases for a singularity, ones involving what we might call "full enmeshment" rather than "partial enmeshment." The argument requires the simulated paradises to be completely immersive, rather than experienced intermittently alongside non-simulated environments and interactions. This does limit the scope of the argument. But it also tells us something about the comparative values of these versions of a good case singularity: assuming the transcendental response to the sceptic is correct, there is a dropoff in subjective quality once human enmeshment in a simulation becomes full (because there is a dropoff in subjectivity simpliciter). This affects rankings in desirability among such options in ways one might not antecedently expect.
Residual sapience — In discussing “brain in a vat scenarios” similar to this one, McDowell includes an interesting aside. For reasons by now familiar, he concludes that “we ought to be able to see that the sheer fact that a brain is going through the motions that an embodied brain goes through when a person thinks or experiences is by itself no ground at all for supposing that there is a mind in there.” Parenthetically, however, he adds that
“It may be a different matter if the mad scientist’s vat contains what used to be an embodied brain; perhaps memory can give subjectivity a tenuous foothold there.” (McDowell, 1994b)
There is some basis for taking it to be indeed a different matter. We experience something approaching Descartes’ plight whenever we dream: despite undergoing (what are presumably) similar internal neurological processes as we actually undergo in events like those depicted in a dream, we are for the time being doing so in a way mostly cut off from our environment. Our dream experience in the main only links up to competently inhabiting our environment by its connection to past and future waking states. This is brought into especially clear relief in the case of vivid dreams among the long term comatose. And yet surely in dreams we have beliefs and experiences with representational content.
Does this offer solace to our future selves enmeshed in simulated paradises? I am not sure. For one, the timescales involved are imagined to be very long, far moreso than any historical episode of dreaming. This may be long enough, and the changes undergone by the “focal user” radical enough, that they might undermine the tenuous foothold McDowell mentions. There is also the fact that proponents of these utopian visions often imagine, not just present‑day humans experiencing these simulations for a very long time, but very large numbers of new beings doing so alongside them. This is often an important component of the visions’ appeal. Whatever consolation is offered by our past foothold in environmental inhabitation will not extend to them. In any event, things look a lot sketchier for the supposed beneficiaries of the new order than tends to be advertised.
Naturalism and zombies — Positions on the nature of experience that enable the transcendental response to go through conflict with very popular beliefs among proponents of these utopian visions in the philosophy of mind. They imply, for example, the possibility of “zombies,” beings internally physically indistinguishable from ourselves that are nevertheless not conscious. My argument may therefore only show that ways of thinking about minds that diverge early on differ also way down the line.
This may be so. But the two pictures of mental content I have sketched are not, on their face, supernaturalist or religious or dualist. Indeed, both McDowell and Brandom go to great lengths to assure readers of their naturalist credentials. They do not invoke spooky immaterial substances or processes, just ordinary physical persons enmeshed in either a larger physical environment or a set of norm‑governed social practices. It follows from their views that fixing the internal physical facts about a brain, say, does not fix what mental states (if any) that brain has. But they are fully consistent with the view, indeed they practically invite it, that fixing all the physical facts whatsoever also fixes all the mental facts. And, as I have indicated, there is nothing in the broad strokes of their views unfriendly to AI sapience. So, while their views on minds might differ in fundamental ways from those of figures like the AI researcher Elmore mentions, they do not require one to take seriously spiritualistic or anthropocentric positions on mindedness.
There is another dimension along which these views differ from some found among optimistic singularitarians, which I should also flag. My argument tacitly assumes at points that representational states and episodes are the be all and end all of consciousness and mindedness. To have a mind is to have states or undergo episodes that are about something, in the sense of representing the world as being a certain way. Aboutness, as the slogan goes, is the mark of the mental.[9] If you take seriously the existence of thoroughly non‑representational mental entities (“qualia,” say), these arguments are likely to seem less persuasive.
Conclusion
I mentioned in the introduction that one could read the preceding argument in one of two ways. The first is the one I myself favour, that therapeutic strategies for assuaging the worries of the sceptic give us significant reason to fear what is sometimes promised as the “good case” of a singularity gone right. The other goes the opposite direction, reasoning backwards from the innocuousness or desirability of such a future to the inadequacy of certain replies to the Cartesian sceptical challenge.
This latter reading gives somewhat less monumental lessons about the best direction for humanity’s conduct. But even so, it underscores the philosophical significance of the singularitarian project. There are a host of stock concerns proponents of such utopian visions for AI are used to acknowledging. What the argument here underscores is that the implications of these projects for traditional philosophical questions are far more sweeping and unpredictable than one might otherwise expect. And it provides philosophers themselves with reason to take such projects seriously as bearing on their own work. If the viability of certain techno‑utopias requires us to abandon seemingly unrelated arguments in the study of scepticism, what else might they upend?
References
Brandom, Robert (1994). Making It Explicit: Reasoning, Representing, and Discursive Commitment. Harvard University Press.
Brandom, Robert B. (2000). Articulating Reasons: An Introduction to Inferentialism. Harvard University Press.
Brandom, Robert (2019). A Spirit of Trust: A Reading of Hegel’s Phenomenology. The Belknap Press of Harvard University Press.
Brentano, F. (1995). Psychology from an Empirical Standpoint.
Routledge, London. Original work published 1874.
Chalmers, David J. (2005). “The Matrix as Metaphysics”. In C. Grau (ed.), Philosophers Explore the Matrix, pp. 132–176. Oxford University Press.
Conant, James (2012). “Two Varieties of Scepticism”. In G. Abel and J. Conant (eds.), Rethinking Epistemology, Volume 2. De Gruyter.
If you do not like the literary device of the evil genius, you can replace it with talk of a Boltzmann Descartes spontaneously generating in the void from statistical noise. ↩︎
For longer and shorter book‑length expositions of Brandom’s views see Brandom, 1994 and Brandom, 2000. The classic source for McDowell’s picture is McDowell, 1994a, but much shorter, clearer, and to my mind more immediately appropriate is McDowell, 1994b. ↩︎
“The Borg does not care that unassimilated beings do not want to be assimilated, and are terrified of it, because once the Borg rewrites them, they will want to be Borg. Honestly, a lot of Singularity talk strikes me the same way. I used to hear my transhuman religionist friends say things to the effect that anyone who was smart enough would understand that, in the Singularity, we would just solve all the problems that people thought they had with the Singularity. That, basically because of that, they didn’t have to take a lot of critique to heart or worry about the consent of everyone else on Earth.” (Elmore, 2026b) ↩︎
My understanding is that this is most developed in print in Brandom, 2019. ↩︎
Something like this seems to be Chalmers’ response to Matrix‑like sceptical scenarios (see Chalmers, 2005). ↩︎
The slogan is a paraphrase of Brentano, 1995. "Every mental phenomenon is characterized by what the Scholastics of the Middle Ages called the intentional (or mental) inexistence of an object, and what we might call, though not wholly unambiguously, reference to a content, direction toward an object (which is not to be understood here as meaning a thing), or immanent objectivity." ↩︎
This essay was mostly written for my own benefit. In style, it is somewhere between a blog post and an article manuscript. In content, it is somewhere between an argument and a meditation. The point was to think through a discomfort about some utopian visions for a “good case” singularity, a discomfort I had not seen explicitly discussed before. Taking all the concerns involved seriously may require an unusual combination of beliefs and sympathies, and so the essay may be of little use to others besides myself. I am putting it here in public, however, in the hope others might at least find it interesting.
In what follows, I am going to argue that a certain collection of philosophical views bears heavily and worryingly on questions about singularitarian artificial intelligence, consciousness or the nature of mind, and long term human survival. Stated so abstractly, the conclusion is neither novel nor particularly controversial — even if the choice of philosophical views and their putative consequences shift from individual to individual. But the specific connection I have in mind has not, to the best of my knowledge, been publicly articulated before. The worries involved, in particular, are largely independent of other worries in the same apparent vicinity.
Here are some more familiar concerns often thought to be raised by certain philosophical views in connection with the aforementioned topics:
These are all important questions, and with the exception of (1) all quite difficult and deeply empirical. But they are not my own question, and do not address my own worry. It comes closest to a mixture of (3) and (5): If we get what sometimes is offered as the ideal outcome of singularitarian AI, would it leave humans conscious or minded in the long run? I will argue that, if we accept a certain family of views about scepticism and mental content, the answer is “No”. These views imply that, in what is sometimes presented as the “best case” of singularitarian AI, humans would not in the long run have any experiences or thoughts — a kind of mental death.
This conclusion does not mean in fact accepting those views. It is conditional: accept the views, the sad answer follows. I myself am both somewhat friendly to the family of views in question and open to the negative answer, and so inclined to derive the negative answer by modus ponens. But one could equally take that answer to be so implausible that we should instead, by modus tollens, reject the family of views. All I mean to establish is the logical connection itself.
The remainder of the current essay lays out the case for this unhappy conclusion. Most of it is expository, summing up things others have said. The original contribution is towards the end, where I put these together to derive the pessimistic conclusion just alluded to.
Dissolving a Philosophical Fear
I will begin with the philosophical ingredients to the argument. There are three: first, a familiar philosophical worry about knowledge and illusion; second, a schema of a strategy for dissolving this worry; third, a couple of ways to instantiate the schema. None are new to me, and my job will be mainly to indicate rather than justify them.
Descartes’ Genius
Consider Descartes. He believes himself to clothed, by the fire in his bedroom, writing a book by candlelight, having planned to do so the whole day preceding. He also believes himself to have several friends, with whom he plans on discussing the book once it is written. To all appearances available to him, that is all indeed the case. It comports with what he (seemingly) remembers, sees, feels, and otherwise can tell about his situation.
However, he is mistaken. He was instead conjured into being in an otherwise uninhabited void moments earlier by an evil demon, or genius. The genius plays upon his senses and memory (that is, causes the appropriate neurons to fire in the memory‑related and sense‑related parts of the brain) using her supernatural powers to falsely persuade him that he is clothed, illuminated, in his bedroom, writing a book, etc. [2] He is unaware of her existence, and aside from this all-encompassing illusion has no interaction with her, and certainly no conversations or social ties.
Descartes’ plight is often taken to present a worry about our supposed knowledge of the world around us. As I prefer to think of this worry, it has two steps for the worried reasoner. First concerns Descartes’ own position. A human in Descartes’ position (summoned into being moments before, totally isolated and unaccompanied aside from the genius playing tricks upon his senses and memory) clearly lacks knowledge of his environment or world. While he believes many things about the world and his place in it, all of which seem to be the case, none of these are in fact the case. He is both ignorant and deluded, because he is in a sceptical genius scenario.
The second step brings the reasoner itself into the picture. It reflects that it has no evidence ruling out its situation being just like Descartes’. It thinks not, of course, but thinking does not make it so. And, if it has no such evidence, how can it know that its circumstances are not just like Descartes’, even if they in fact are not? Its previous conclusion about Descartes’ ignorance, that is, leads it to worry that (appearances and convictions aside) it too is ignorant, even if its situation is not otherwise like his (newly sentient, deceived by a demon, etc.) It is (it fears) ignorant, even if not deluded, because it cannot rule out that it might be in a sceptical genius scenario.
This presentation of the worry has the advantage of isolating two separate features of the puzzle. One is the first-personal, subjective component: the reasoner’s worry about its own knowledge and ignorance. But distinct from this, and more fundamental to the chain of reasoning, is a third-personal, removed assessment of Descartes’ condition as apart from its own. It is by being convinced of Descartes’ would‑be credulous ignorance that it comes to fear its actual credulous ignorance. This distinction, and order of priority, will help clarify the resolution we are about to consider.
The Transcendental Escape
There are several ways of trying to address the sceptical reasoner’s worries about its state of knowledge. Some argue, essentially, that its deduction from Descartes’ ignorance to its own is fallacious, or otherwise should not convince it. The strategy I am interested in here, however, tries to undercut the reasoning before it ever gets that far, by questioning the conclusion the reasoner draws about Descartes’, before it starts considering itself and its position in relation to his. The strategy, sometimes called a transcendental or Kantian way of addressing sceptical worries, is especially associated with James Conant. [3]
We described Descartes, just above, as having been mistaken and subject to delusion about his true situation. We also described him as having been in a sceptical genius scenario: just sprung into existence, his faculties of sense and memory operating at total disconnect from his actual environment. From this, we (and the reasoner) conclude that he is ignorant of his true situation. These three all go together naturally: the description of the sceptical genius scenario lends apparent credibility to the claims about his being mistaken, illuded, and ignorant.
The proponent of the transcendental approach, however, disagrees. A being in Descartes’ position cannot be subject to error or illusion, because to suffer error or illusion is to somehow represent the world (or have the world represented to one) otherwise than as it is, for example by representing it as containing a fire, one’s bedroom, etc. It involves having mental states or acts with contents that are false. A being in Descartes’ position, however, does not represent the world in any way, let alone otherwise than as it is. It has no contentful states or acts, and a fortiori no false ones. While Descartes does not know anything about the world, neither is Descartes ignorant of it, any more than a star or water droplet is ignorant. An appropriate analysis of mental representation shows that such a being’s situation is too deprived, adverse, and spare to even rise to that level.
It is worth getting somewhat clear on what we mean by “Descartes’ position”. It cannot mean, according to the proponent of the transcendental response, his position as described. According to them, after all, that description contains a contradiction. Instead, we start with an incoherent description (Descartes believing himself to be clothed, befriended, etc. but in fact deceived by the genius), and then once the incoherence has been exposed retreat to a weaker, but coherent condition (just as described above, but minus the representational states attributed to Descartes). It is a procedure of moving from an easy‑to‑convey but contradictory state to a coherent but harder to explicitly specify state indicated by “scaling back” from the source of the contradiction.
This response stops the worried reasoner at the root. The reasoner begins to worry that what it believes and how things seem to it are much like what Descartes believes and how things seem to him, and thence is led to suspect it is broadly ignorant. But Descartes, like anyone (anything?) in a sceptical genius scenario, in fact believes nothing and has nothing seem to him in any way. The sceptical reasoner might, of course, start to more radically doubt its own mindedness, its own ability to represent the world at all. But Descartes’ situation gives no basis for believing in the halfway house the reasoner feared: that it believes things to be a certain way but also that these beliefs do not rise to the level of knowledge.
Two Pictures
The preceding argumentative strategy is presented schematically. It leaves the main step unspecified: the analysis of mental representation on which a being like Descartes cannot mentally represent.
There are several different such analyses that have been taken (by their proponents or others) to fill this role. Here I will briefly and roughly describe two, both associated with major figures from the University of Pittsburgh: one, from Robert Brandom, on which socially instituted normative statuses play a central role in the possession of mental content; another, from John McDowell, on which coping with and taking in the world about one plays a similarly central role in the possession of mental content. [4]
Brandom’s Picture
To have attitudes (belief, knowledge, desire, fear, subjective appearance, etc.) with specific contents (that one is clothed and illuminated, that one might be alone and deluded, etc.) depends upon one’s involvement in a larger social practice. This is not a contingent physical fact about the social infrastructure required for survival, education, or the like, but an essential fact about mental content as such.
These mental states and acts play an ultimately auxiliary role in an agent’s representing the world. What is fundamental is their undertaking certain commitments and enjoying certain entitlements in the practice of giving and asking for reasons. In asserting that a shirt is olive, say, one undertakes a commitment to its being green, a commitment that precludes entitlement to claiming it is red. This practice is inherently social: one undertakes these commitments to others also doing so, and enjoys such entitlements from them. We attribute other mental states to one another, roughly, as part of tracking their commitments and entitlements among one another. Apart from the great game of giving and asking for reasons, there is no role for belief, knowledge, desire, etc. to play.
This picture (here sketched only very crudely) is clearly suited to play the desired role in the transcendental response to the sceptical reasoner. Descartes is in no position to give or ask reasons to or from anyone. He is asocial and, therefore, amental.
It is worth noting that there is nothing inimical to AI sapience (mental representation) in Brandom’s picture. We attribute contentful attitudes to one another to track us as players in the practice of giving and asking for reasons, not to recognise some special biological or spiritual quality in those to whom we attribute the attitudes. There is no more an impediment to attributing such attitudes to machines than there is to integrating them into that practice.
McDowell’s Picture
In good cases of perceiving and acting, there is no distinction between the mental contents of an agent and the worldly states they represent. In a case of genuine perception, an agent is simply open to whatever it is about the world that she perceives; she “takes in” that, say, the snow is melting before her, and this aspect of the world just is her perceptual content, what she sees. In a case of intentional action, say intentionally painting her door white, there is similarly no distance between the content of her intention and what she achieves in the world. The fact achieved, like the fact taken in, is itself the content of the successful intention, or veridical perception.
Not all appearances or intentions are happy in this way. Sometimes one merely seems to see something one does not in fact see, as when in the Müller‑Lyer illusion one seems to see that two lines, in fact of equal length, are of different lengths. Sometimes one merely intends an action one does not realise, as when a drunk intends to walk in a straight line but cannot. But these are parasitic upon cases of successful perception and action: they are identifiable as failures only against the backdrop of successes. They are failed exercises of fallible capacities. Something incapable of veridical perception or successful action, say a thing embedded in an environment totally hostile to such, would lack even the capacity, and thus be able to exercise it neither in success nor in failure.
Perception and action are the bedrock of mental representation. Absent the capacity for either, thought cannot make contact with the world, without which there is nothing for it to be thought about. Thus McDowell, too, fits in with the transcendental response to the sceptical reasoner. Descartes is so incapable of taking in or acting upon the world that he cannot represent it at all, nor suffer from illusions with misleading contents.
Note that McDowell’s picture, like Brandom’s, has room for perception, action, and thought in AI. McDowell is quite at pains to assure the reader that there is ample room in his story for the work of cognitive science in uncovering the physical mechanisms underwriting an agent’s ability to take in the world; there is nothing in his story requiring these mechanisms to be biological rather than artificial.
Let us sum up. In the first of the three points we considered, we saw how a reasoner considering the situation of Descartes might be led, first, to credit him with extreme deluded ignorance and, second, to draw pessimistic conclusions about its own knowledge of the world in light of that. We then saw one strategy for defusing this worry, which would tell the reasoner not even to credit Descartes with the minimally sapient condition of total delusion. Finally, we saw how Brandom and McDowell’s stories about the point of mental representation in the life of thinking beings can offer support to this strategy.
I would again like to disclaim any endorsement of Brandom or McDowell on these two pictures. A full articulation, much less a defence, of either falls well out of scope, and I am not sure the two pictures are even jointly consistent. I am, personally, much more sympathetic to the latter than the former. The point is simply to observe that they exist, and to ask what conclusions they might imply about our guiding question.
Proposing a Technological Hope
Holly Elmore, a very vocal critic of current AI research, recounts a conversation with an AI engineer:
This vision of a Universal Basic Pleasure‑Palace is extreme. But it is a natural end point of a common family of techno‑utopian aspiration. Elmore goes on to cite Nick Bostrom and Will MacAskill, both prominent figures in the intellectual milieu around frontier AI development, who offer similar (if not always quite so grandiose) visions of an optimistic AI future.
The comparison with the Matrix is apposite. Just as the Matrix offers a vision of a collective sceptical genius scenario, so the hopeful picture here on offer promises a world of individualised kindly genius scenarios. As Descartes’ genius pulls the wool over his eyes in a campaign of systematic all‑encompassing deception, so the friendly AI in this researcher’s conception spins up a full‑blown simulated reality to cater to its human beneficiary’s every whim.
Elmore herself regards this prospect with obvious and intense contempt. Here she is concerned about whether human beings might disapprove of the uses to which their simulacra are put in these digital Xanadus. Elsewhere she protests that it might not even be what their supposed main beneficiaries want, at least not presently: does the man on the street really dream of being separated from his friends and family, so they can each be cast the starring role in a simulated reality pandering to their least desire and peopled by replicas of their former loved ones? [6]
The natural retort to this complaint is that it is by the post‑singularity lights of the man on the street that we ought to judge his desires for the singularity, just as it is by the replicated loved ones’ desires we ought to gauge their consent. Elmore is quite contemptuous of this response, too, but it is hard to say what exactly is so unjust about the proposal. I can come up with no compelling argument that the man in the street would take poorly to the envisioned future of personalised utopias. And often it does make sense to shunt people into situations they might initially dread but in the moment will appreciate — this is much of the task of raising a child into adulthood. Why not defer to the wishes of their post‑singularity selves? This is not entirely rhetorical; I am genuinely unclear on the ethics involved.
Kindly Geniuses in a Data Centre
I hope that my concern about these utopian visions should now be obvious. It is not that some people currently do not want them — it may be better to defer on this matter to their future desires, which for all we have seen might be more accommodating. It is also not that, in the Matrix‑like utopia envisioned, the recipients of the paradisaical digital universes would be unhappy with them, or not want them. It is that, if the transcendental approach to the sceptic is right, they would not want or be unhappy about anything at all. Their situation would be like that of Descartes: not of being deluded against his will, but of lacking delusion or will altogether.
No Representation in Paradise
Consider first Brandom’s picture. It is an important part of his overarching view of human reasoning, which has become only more prominent as his work has progressed, that the practice of giving and asking for reasons essentially involves the rectification of improprieties according to the norms of the practice: relinquishing commitments to which those engaged in the practice are not entitled, newly making good on the conversational entitlements of practitioners, and so on. [7] It requires, in short, the sorts of conversational friction and disambiguation that my maximally indulgent coterie of simulated attendants is supposed to rule out. And, on his view, as my part in the practice of giving and asking for reasons goes, so goes my capacity for thought.
Next consider McDowell’s picture. The bedrock of mental representation, as I have said, is for McDowell the successful taking in of the world through perception, and the successful achievement of worldly facts in intentional action. Other mental representation enjoys that status through its connection to these two basic operations. But perception and action, too, are intelligible as perception and action only by the role they play in accommodating the agent to the world surrounding it, what McDowell calls “more or less competently inhabiting an environment.” In the case of a frog, this consists in catching insects, avoiding predators, and in general coping with its surroundings to survive and flourish — an idea McDowell takes explicitly from J. J. Gibson.
Again, this fits at best only awkwardly into the picture of the unnamed AI engineer from Elmore’s story. What would transpire in a pleasure palace “came down to what the focal user wanted in that instance.” This does not describe an environment with which to cope so much as a dream through which to float. It is no part of McDowell’s account, of course, to deny the existence of dreams, or that they possess content as illusions or hallucinations. But, without a non‑dreamlike environment to serve as their backdrop, they provide no means to prevent a “frictionless spinning in the void” of the dreaming subject. That is, they do not by themselves offer what McDowell would accept as enough mental contact with reality for thought to take on representational content.
Finally, it is worth distinguishing this general line of argument from worries about “meaning” or “purpose” for humans in a world where their labour is rendered obsolete. Consider McDowell’s case of the frog, who is able to represent its environment because it is engaged in the overarching activity of competently inhabiting it. One can, if one likes, call this the “meaning” or “purpose” of the frog’s life. But note how little contact meaning and purpose, so conceived, have with the human anxieties usually to attend them. Would a man, in the throes of a mid‑life crisis, take solace in the knowledge that his life has meaning in “competently inhabiting” his “environment”? If initially satisfied, would this persist if he learned that no more was being said of him than could also be said of a frog to spend its life patiently watching for flies to eat and snapping turtles to hide or flee from?
Worries about meaning, in this august sense, after the singularity miss the mark by starting too high. It may be that it is important to us for our lives to have a large, all‑encompassing sort of meaning, of life projects that genuinely contribute to a larger community or historical task. But if so the pursuit of such projects and tasks must involve many cases of meaning on a much smaller, more mundane level: instances of representing the world in belief, desire, and perceptual appearance. If a singularitarian intelligence threatens these more basic bits of meaning, worries about meaning of the larger sort become moot. The more fundamental threat is not of life’s crowning glory being deprived us but of our undergoing a (very resource‑intensive) sort of cognitive euthanasia.
Caveats
There are several caveats worth flagging about this train of thought.
Simulated reality — It is open, of course, for proponents of the vision to protest that there is no reason not to accept these individualised pleasure palaces as genuine environments for their “focal users” in which to engage in the practice of giving and asking for reasons, or whatever else might be required to endow thought with content. [8]
But notice how poorly this fits with the transcendental response! If the utopian claims that the simulated persons and layout of the focal user’s paradise suffice to render their thoughts and appearances contentful, why not say the same of Descartes’ hallucinated room and companions? This retort, if they take it, robs the transcendental response of its purchase — or, contraposing, if that response to the sceptical reasoner has purchase, then the utopian cannot offer that retort.
The argument also targets a fairly specific range of optimistic cases for a singularity, ones involving what we might call "full enmeshment" rather than "partial enmeshment." The argument requires the simulated paradises to be completely immersive, rather than experienced intermittently alongside non-simulated environments and interactions. This does limit the scope of the argument. But it also tells us something about the comparative values of these versions of a good case singularity: assuming the transcendental response to the sceptic is correct, there is a dropoff in subjective quality once human enmeshment in a simulation becomes full (because there is a dropoff in subjectivity simpliciter). This affects rankings in desirability among such options in ways one might not antecedently expect.
Residual sapience — In discussing “brain in a vat scenarios” similar to this one, McDowell includes an interesting aside. For reasons by now familiar, he concludes that “we ought to be able to see that the sheer fact that a brain is going through the motions that an embodied brain goes through when a person thinks or experiences is by itself no ground at all for supposing that there is a mind in there.” Parenthetically, however, he adds that
There is some basis for taking it to be indeed a different matter. We experience something approaching Descartes’ plight whenever we dream: despite undergoing (what are presumably) similar internal neurological processes as we actually undergo in events like those depicted in a dream, we are for the time being doing so in a way mostly cut off from our environment. Our dream experience in the main only links up to competently inhabiting our environment by its connection to past and future waking states. This is brought into especially clear relief in the case of vivid dreams among the long term comatose. And yet surely in dreams we have beliefs and experiences with representational content.
Does this offer solace to our future selves enmeshed in simulated paradises? I am not sure. For one, the timescales involved are imagined to be very long, far moreso than any historical episode of dreaming. This may be long enough, and the changes undergone by the “focal user” radical enough, that they might undermine the tenuous foothold McDowell mentions. There is also the fact that proponents of these utopian visions often imagine, not just present‑day humans experiencing these simulations for a very long time, but very large numbers of new beings doing so alongside them. This is often an important component of the visions’ appeal. Whatever consolation is offered by our past foothold in environmental inhabitation will not extend to them. In any event, things look a lot sketchier for the supposed beneficiaries of the new order than tends to be advertised.
Naturalism and zombies — Positions on the nature of experience that enable the transcendental response to go through conflict with very popular beliefs among proponents of these utopian visions in the philosophy of mind. They imply, for example, the possibility of “zombies,” beings internally physically indistinguishable from ourselves that are nevertheless not conscious. My argument may therefore only show that ways of thinking about minds that diverge early on differ also way down the line.
This may be so. But the two pictures of mental content I have sketched are not, on their face, supernaturalist or religious or dualist. Indeed, both McDowell and Brandom go to great lengths to assure readers of their naturalist credentials. They do not invoke spooky immaterial substances or processes, just ordinary physical persons enmeshed in either a larger physical environment or a set of norm‑governed social practices. It follows from their views that fixing the internal physical facts about a brain, say, does not fix what mental states (if any) that brain has. But they are fully consistent with the view, indeed they practically invite it, that fixing all the physical facts whatsoever also fixes all the mental facts. And, as I have indicated, there is nothing in the broad strokes of their views unfriendly to AI sapience. So, while their views on minds might differ in fundamental ways from those of figures like the AI researcher Elmore mentions, they do not require one to take seriously spiritualistic or anthropocentric positions on mindedness.
There is another dimension along which these views differ from some found among optimistic singularitarians, which I should also flag. My argument tacitly assumes at points that representational states and episodes are the be all and end all of consciousness and mindedness. To have a mind is to have states or undergo episodes that are about something, in the sense of representing the world as being a certain way. Aboutness, as the slogan goes, is the mark of the mental. [9] If you take seriously the existence of thoroughly non‑representational mental entities (“qualia,” say), these arguments are likely to seem less persuasive.
Conclusion
I mentioned in the introduction that one could read the preceding argument in one of two ways. The first is the one I myself favour, that therapeutic strategies for assuaging the worries of the sceptic give us significant reason to fear what is sometimes promised as the “good case” of a singularity gone right. The other goes the opposite direction, reasoning backwards from the innocuousness or desirability of such a future to the inadequacy of certain replies to the Cartesian sceptical challenge.
This latter reading gives somewhat less monumental lessons about the best direction for humanity’s conduct. But even so, it underscores the philosophical significance of the singularitarian project. There are a host of stock concerns proponents of such utopian visions for AI are used to acknowledging. What the argument here underscores is that the implications of these projects for traditional philosophical questions are far more sweeping and unpredictable than one might otherwise expect. And it provides philosophers themselves with reason to take such projects seriously as bearing on their own work. If the viability of certain techno‑utopias requires us to abandon seemingly unrelated arguments in the study of scepticism, what else might they upend?
References
Answer: Of course! Why could they not? ↩︎
If you do not like the literary device of the evil genius, you can replace it with talk of a Boltzmann Descartes spontaneously generating in the void from statistical noise. ↩︎
For the classic exposition see Conant, 2012. ↩︎
For longer and shorter book‑length expositions of Brandom’s views see Brandom, 1994 and Brandom, 2000. The classic source for McDowell’s picture is McDowell, 1994a, but much shorter, clearer, and to my mind more immediately appropriate is McDowell, 1994b. ↩︎
Elmore, 2026a ↩︎
“The Borg does not care that unassimilated beings do not want to be assimilated, and are terrified of it, because once the Borg rewrites them, they will want to be Borg. Honestly, a lot of Singularity talk strikes me the same way. I used to hear my transhuman religionist friends say things to the effect that anyone who was smart enough would understand that, in the Singularity, we would just solve all the problems that people thought they had with the Singularity. That, basically because of that, they didn’t have to take a lot of critique to heart or worry about the consent of everyone else on Earth.” (Elmore, 2026b) ↩︎
My understanding is that this is most developed in print in Brandom, 2019. ↩︎
Something like this seems to be Chalmers’ response to Matrix‑like sceptical scenarios (see Chalmers, 2005). ↩︎
The slogan is a paraphrase of Brentano, 1995. "Every mental phenomenon is characterized by what the Scholastics of the Middle Ages called the intentional (or mental) inexistence of an object, and what we might call, though not wholly unambiguously, reference to a content, direction toward an object (which is not to be understood here as meaning a thing), or immanent objectivity." ↩︎