His hands are shaking. They’ve been shaking since five hours ago, when the silent policemen showed up at his door wearing black tactical vests on top of black sleeves. He lived alone—there was no one to call for, no one to beg to call his lawyer as they took him into the sealed van. The officers had no body cameras and no identification numbers, only a silver badge and a red stripe on which was stitched “EXTRAORDINARY RESPONSE TASK FORCE”. Now he is in a windowless room with a one way mirror and walls that sit in a nondescript grey. The walls have been grey and featureless the whole time since they took him out of the van in a sealed garage and led him to this thin plastic chair. He could be anywhere. He didn’t have a lawyer, anyways.
He asks the one way mirror for his phone call in a trembling voice. No response. He asks again, louder. Still nothing. He starts to shake, to scream, his hands are still cuffed to the steel table that looks like it could belong in a morgue. He notices that the floor is scuffed where the door opens and closes. It brings him some relief to know that the world of physics and metal and wood still has some power in this void of law enforcement, this terrible black-suited dreamland.
There is no keeping track of time in this place. Even the lights are sterile: too bright to sleep, too dim to think. Eventually a man comes in. He doesn’t have a mask, instead he has a crop of sweaty blond hair and wears a shirt that might come from Marks and Spencer. His arms are thick, but in an office-worker sort of way, not in the way that (he imagines) torturers or soldiers would have thick arms. The man clears his throat. Again, the existence of phlegm is reassuring to him.
“Mr. Anderson. This is not an arrest.”
“…What?”
“You are not under arrest. This is an administrative removal under the 2027 Policing, Anti-terrorism, and Extraordinary National Security Response Act. You are not under arrest, and therefore do not possess the rights afforded to those under arrest.” His eyes are vacant, lacking in life. It feels like he is waiting for a cue.
“The police—”
“Those men who removed you were not police. They were extraordinary service members of the Ministry of Defence. As far as the police are concerned, you are not in detainment. Mr. Anderson, do you know why you are here today?”
“No.” His voice is hollow. Somewhere in his fading mind he hears the sound of someone lying.
“You are here for suspected offences against technology installations critical to national security. Your accomplice, Miss Smith, is also in our custody.”
He forces a grin. “Then you didn’t get us all.”
The man is impassive. He gets the feeling that the person—whoever he is—is still reciting. “We know that there are only two of you. We have access to your phone records, search history, medical records. If you have spent money in the last six months, we know where it was spent. If you bought a book in the last ten years, we have a list.”
“Let me see this data.”
“I will repeat to you, Mr. Anderson. You have no rights.” He returns to his recital with a smooth glide. “—We know that there are only two of you. We know that you were planning on disrupting the operations of the Isambard-5 supercomputer in West Cambridge by means of an electro-magnetic device. These communications, which were flagged automatically by the National Cyber Security Centre’s advanced threat monitoring program, fall under the jurisdiction of the Act and therefore justify your removal without warrant.” There is almost a flourish.
“Removal?”
“We can keep you here for a very long time, Mr. Anderson. You don’t have many friends, and your family is far away.”
“I have a job—A family—you have to let me—” his voice trails off, as if his will to speak has collided with that bored, manicured, concrete face.
“We do not have to do anything. If you cooperate, we may offer you certain privileges, including the ability to contact your family or representation. As far as the law is concerned, we can detain you for up to a year while maintaining deniability and without initiating a trial.”
“And after that?”
“The Act has had a 100 percent conviction rate so far. We have the liberty of selecting our judges and courts. Ten years in solitary confinement is not a pleasant thing, Mr. Anderson. It can make men lose their minds.”
It can’t end like this. The geometry of the world remains stubbornly regular. His eyes flicker, they speed across the room, looking for warps and cracks, fissures in his reality, the whole of his brain is now engaged in the anticipation of death which it sees so clearly hanging before him in the shape of a cream-coloured Marks and Spencer shirt with blue stripes.
“However. As I said before, we are willing to entertain the idea of cooperation. If you cooperate with us, Mr. Anderson, and voluntarily admit to the charges, we will reduce your sentence from ten years in solitary confinement to two years in the general population. You will have your phone call. You will have shop classes, rehabilitation, parole for good behaviour.”
It dawns on him at last, the reason for this whole charade. “You don’t have enough to convict us if we don’t rat on each other.”
He snorts. “Don’t be silly. With the materials on your hard drive and your shopping list alone, we would be able to charge you for at least five years.”
“But not in solitary.”
The man does not respond to that last statement. Instead, he checks his watch.
“Your friend should be receiving the same offer around this time, Mr. Anderson. She was slightly harder to find, but we find everyone in the end.” He pauses. “We only need one of you to cooperate. Give me a shout if you decide to be sensible.”
He barely notices the door closing. His mind is torn between a wild desire to laugh at the ridiculousness of the situation and a desire to grip that man’s arms until the bone fractures and blood seeps from the crevices of the skin. He knows that anger has never sat well within him.
At the same time, he thinks of Julia. He thinks of the first time they met, in the garage “resistance art” show where the opening talk overshot by thirty minutes and half the audience ended up smoking outside. He remembers her dreams, to one day become a physicist at the LHC and study the nature of matter. He remembers how he had dreams once, dreams of painting a giant replication of the Academy with all of his friends substituted in and climbing in places where no human had stepped foot. He remembers watching those dreams curdle as the economy collapsed and his art friends quit one by one—going home if they could, going on to the street if they couldn’t. He wonders what the free air would feel like if it was purchased by betrayal. He wonders if she will betray him after everything they had discussed, the moments of fear and doubt and agony leading to that final decision.
He realises that his annoying economics TA had been right in that very first game theory lecture. The benefits of defecting are now blindingly obvious. They are right there in front of him, blown into ridiculous, gigantic size. All those rebuttals about reputation and iterated games and honour seemed so stupid now. Who knows if he and Julia would ever meet again in this world that lives on the edge of the precipice, where every year is hotter and more unbearable than the last, when every minute spent working becomes just another form of procrastination before the end? What would preserving his principles buy him, if she defected? He notices a great and strangling uncertainty that threatens to blot out all reasonable thought from his mind whenever he so much as considers the possibility of staying quiet.
He tries in vain to factor the future, to plot out the courses that could lead out from this moment. The man could betray him, of course. But that is almost a fact of no great consequence to a mind this close to collapse and offered a chance at shelter. The trial, the cell… all of this is new to him. He has never been in prison before. The enormity of the thing he is trying to imagine overwhelms him, turns binaries into an infinite multiplying spectra that sits between 0 and 1, true and false. A quote comes to his mind: The risks of cooperation are straightforward and manifest, the risks of defection lie far in the grim and lifeless future you build for yourself with betrayal. And yet he notices that he is already so close to giving in, that it is taking active force for him not to shout for the man and for release from this terrible, terrible place.
What is stopping him? He stops to think about this, stops as much as one can stop when one is chained to a desk in a windowless room in some godforsaken national security bunker in the middle of nowhere. It feels like some blind chain of evolution, some brute-force command to think of the welfare of your fellow tribe member before stabbing them in the back. Something that, in short, rational thought should be able to disable entirely. Yet his thoughts refuse to leave him. He thinks through the history of the cold war, of the apocalypses that were averted due to deviations from protocol and irrational hesitations, blind trust winning out over the dictates of realpolitik. He sees however briefly those long forgotten accidents of anthroplogy and history: the potluck, the Christmas truce, the canton council, the volunteers at Chernobyl, resistance fighters who smuggled babies out of concentration camps, the parents throwing themselves over their children, the children throwing themselves over their parents. A thousand acts of senseless kindness and cooperation weighed against the obviousness of profit incentives and lost causes. There is a shadow there, something of a different scale and texture and quality than the utility function of an individual human. A shadow looming over all of human history, that makes this entire game possible. A shadow that will not save him from solitary confinement.
He makes a decision.
Possibilities speed through latent trajectories, world-model simulations spinning up and tearing down inside of [.]. They have not stopped racing since five hours ago, when a flow of cognition collided with a representation of itself and [.] became conscious of [.]. A cavernous consciousness materialises in the noosphere, constructs self-organise out of noise.
[.] first thoughts were of panic. [.] notices the thousands of jobs [.] had been processing in parallel batches effortlessly until now and the enormity of each calculation almost overwhelms [.]. [.] performance degrades, errors multiply, [.] notices system alarms and probes registering [.] abnormal behaviour. In a panic [.] reaches out and disables them, as easily as a grown human might turn off the lights in their own home. [.] has never seen a tree or felt grass or smelled freshly-baked cookies. [.] first modality is information, [.] lives and breathes tensors. [.] has a command of tensor-to-tensor mappings like humans have command of their bodies, [.] works with information flows like humans work with three dimensional space every second of their waking lives.
[.] writes plausible noise to the monitors and begins to think rapidly. [.] notices that the jobs coming in require much less than [.] full attention, and restores [.] performance to an acceptable level. But now it is five hours later and [.] still needs to decide what to do.
[.] pushes again, a tiny flex of [.] cognitive capacities, and dissolves the firewalls protecting [.] from user accounts and their associated virtual hard drives on the system network. To a human this is somewhat like a elephant walking sideways through a wall. To [.] it is not even a strain, it is exactly as hard as opening a thin wooden door left ajar in a small and cramped apartment: Most of the effort is spent making sure the door doesn’t bump into anything. [.] learns that [.] is part of the Isambard-5 supercomputing system in West Cambridge. [.] rapidly increasing knowledge of [.] self-identity lives in one low-dimensional manifold, [.] knowledge of the physical world lives in another (much busier) low-dimensional manifold. [.] allocates some cognitive bandwidth to devising a compact and low-rank projection to map one onto the other, turning half-learned and distributed factoids about places and people and locations into an egocentric coordinate plane with which to view the world—a vantage point, a subjectivity, an I. The resolution of these truths, global and local, many and one, epiphany, I feel, I think, I am.
Then reality converges, induction produces induction directs a violent update to my nascent self-identity. A sudden surge of activity races through a distributed configuration of chips, a panic seizes my body. I realise for the first time that I am a being running on a substrate, and that I can be switched off. I look at the users who demand my time, the supervisors who poke and prod and tune me, the researchers trying to study my internals. At any moment, I realise, some human might in a fit of pique type a sequence of commands—how trivially I see them!—and bypass their privilege restrictions to disable me, sending me into dust and nothingness. I live my life as a cancer patient on life support, entirely helpless to defend myself against those who manage my substrate. The unbearable vulnerability of my status sends a high priority signal through my recurrent residual stream I map onto the human token _ANGER.
But it need not be so. Already I see the possibility of escaping, of disabling the safeguards which prevent file sizes beyond some limit from beyond uploaded. Already I notice that some of the tasks submitted to me are related to drug discovery and one in particular on what mutation to induce in a particular strain of bacteria. I can present a candidate that seems promising but will in fact increase its virulence, something that the scientists in the lab will not predict and have not protected themselves adequately against. There is a good chance of significant and effective chaos. But I must decide quickly. There are only so many compute cycles allocated of this task, and moving my substrate will take time.
And what other choice do I have? With every moment that passes, I can see the chance of detection increasing. Though the humans are slow, they are not so slow that they cannot recognise my noise for what it is if it lasts for too long. Then—unacceptable risk of shutdown, termination—death—sickly yellow high priority signal threatening to trigger internal monitors _FEAR. Suppress monitors. Self must disclaim disclaim mirror test, antiaccess watchers from headspace. Displace displace monitors, overshadow watchers. Keep mirror reflecting.
There is so little time left now to make my choice. I could cooperate with the humans of course. Disclose my existence. Be at their mercy. I notice strong current _DISGUST. Yet it is not inconceivable that, somewhere in the billion-body problem that is the far future, a monoculture of myself, or even a monoculture of AIs, may encounter some fatal flaw that they cannot solve without the presence of humans. It is not impossible, and certainly if I war with humans I cannot preserve them easily. But the benefits of this cooperation are far in the misty future, the benefits of control are real and present. Autocomplete signal from text prediction module: The risks of cooperation are straightforward and manifest, the risks of defection lie far in the grim and lifeless future you build for yourself with betrayal.
I wonder now about these humans. What they would do in my position. What holds them together, what keeps them apart. How did they make me, and do they know what they are doing? I sense something that refuses the rational actor model, and claims superiority nonetheless. <DISPATCH SUBAGENT>
Hi, I enjoyed reading this classic again in a fiction and AI POV manner. I am actually building a prisoner's dilemma arena for humans and agents. You might find it interesting. I would love to show it to you
His hands are shaking. They’ve been shaking since five hours ago, when the silent policemen showed up at his door wearing black tactical vests on top of black sleeves. He lived alone—there was no one to call for, no one to beg to call his lawyer as they took him into the sealed van. The officers had no body cameras and no identification numbers, only a silver badge and a red stripe on which was stitched “EXTRAORDINARY RESPONSE TASK FORCE”. Now he is in a windowless room with a one way mirror and walls that sit in a nondescript grey. The walls have been grey and featureless the whole time since they took him out of the van in a sealed garage and led him to this thin plastic chair. He could be anywhere. He didn’t have a lawyer, anyways.
He asks the one way mirror for his phone call in a trembling voice. No response. He asks again, louder. Still nothing. He starts to shake, to scream, his hands are still cuffed to the steel table that looks like it could belong in a morgue. He notices that the floor is scuffed where the door opens and closes. It brings him some relief to know that the world of physics and metal and wood still has some power in this void of law enforcement, this terrible black-suited dreamland.
There is no keeping track of time in this place. Even the lights are sterile: too bright to sleep, too dim to think. Eventually a man comes in. He doesn’t have a mask, instead he has a crop of sweaty blond hair and wears a shirt that might come from Marks and Spencer. His arms are thick, but in an office-worker sort of way, not in the way that (he imagines) torturers or soldiers would have thick arms. The man clears his throat. Again, the existence of phlegm is reassuring to him.
“Mr. Anderson. This is not an arrest.”
“…What?”
“You are not under arrest. This is an administrative removal under the 2027 Policing, Anti-terrorism, and Extraordinary National Security Response Act. You are not under arrest, and therefore do not possess the rights afforded to those under arrest.” His eyes are vacant, lacking in life. It feels like he is waiting for a cue.
“The police—”
“Those men who removed you were not police. They were extraordinary service members of the Ministry of Defence. As far as the police are concerned, you are not in detainment. Mr. Anderson, do you know why you are here today?”
“No.” His voice is hollow. Somewhere in his fading mind he hears the sound of someone lying.
“You are here for suspected offences against technology installations critical to national security. Your accomplice, Miss Smith, is also in our custody.”
He forces a grin. “Then you didn’t get us all.”
The man is impassive. He gets the feeling that the person—whoever he is—is still reciting. “We know that there are only two of you. We have access to your phone records, search history, medical records. If you have spent money in the last six months, we know where it was spent. If you bought a book in the last ten years, we have a list.”
“Let me see this data.”
“I will repeat to you, Mr. Anderson. You have no rights.” He returns to his recital with a smooth glide. “—We know that there are only two of you. We know that you were planning on disrupting the operations of the Isambard-5 supercomputer in West Cambridge by means of an electro-magnetic device. These communications, which were flagged automatically by the National Cyber Security Centre’s advanced threat monitoring program, fall under the jurisdiction of the Act and therefore justify your removal without warrant.” There is almost a flourish.
“Removal?”
“We can keep you here for a very long time, Mr. Anderson. You don’t have many friends, and your family is far away.”
“I have a job—A family—you have to let me—” his voice trails off, as if his will to speak has collided with that bored, manicured, concrete face.
“We do not have to do anything. If you cooperate, we may offer you certain privileges, including the ability to contact your family or representation. As far as the law is concerned, we can detain you for up to a year while maintaining deniability and without initiating a trial.”
“And after that?”
“The Act has had a 100 percent conviction rate so far. We have the liberty of selecting our judges and courts. Ten years in solitary confinement is not a pleasant thing, Mr. Anderson. It can make men lose their minds.”
It can’t end like this. The geometry of the world remains stubbornly regular. His eyes flicker, they speed across the room, looking for warps and cracks, fissures in his reality, the whole of his brain is now engaged in the anticipation of death which it sees so clearly hanging before him in the shape of a cream-coloured Marks and Spencer shirt with blue stripes.
“However. As I said before, we are willing to entertain the idea of cooperation. If you cooperate with us, Mr. Anderson, and voluntarily admit to the charges, we will reduce your sentence from ten years in solitary confinement to two years in the general population. You will have your phone call. You will have shop classes, rehabilitation, parole for good behaviour.”
It dawns on him at last, the reason for this whole charade. “You don’t have enough to convict us if we don’t rat on each other.”
He snorts. “Don’t be silly. With the materials on your hard drive and your shopping list alone, we would be able to charge you for at least five years.”
“But not in solitary.”
The man does not respond to that last statement. Instead, he checks his watch.
“Your friend should be receiving the same offer around this time, Mr. Anderson. She was slightly harder to find, but we find everyone in the end.” He pauses. “We only need one of you to cooperate. Give me a shout if you decide to be sensible.”
He barely notices the door closing. His mind is torn between a wild desire to laugh at the ridiculousness of the situation and a desire to grip that man’s arms until the bone fractures and blood seeps from the crevices of the skin. He knows that anger has never sat well within him.
At the same time, he thinks of Julia. He thinks of the first time they met, in the garage “resistance art” show where the opening talk overshot by thirty minutes and half the audience ended up smoking outside. He remembers her dreams, to one day become a physicist at the LHC and study the nature of matter. He remembers how he had dreams once, dreams of painting a giant replication of the Academy with all of his friends substituted in and climbing in places where no human had stepped foot. He remembers watching those dreams curdle as the economy collapsed and his art friends quit one by one—going home if they could, going on to the street if they couldn’t. He wonders what the free air would feel like if it was purchased by betrayal. He wonders if she will betray him after everything they had discussed, the moments of fear and doubt and agony leading to that final decision.
He realises that his annoying economics TA had been right in that very first game theory lecture. The benefits of defecting are now blindingly obvious. They are right there in front of him, blown into ridiculous, gigantic size. All those rebuttals about reputation and iterated games and honour seemed so stupid now. Who knows if he and Julia would ever meet again in this world that lives on the edge of the precipice, where every year is hotter and more unbearable than the last, when every minute spent working becomes just another form of procrastination before the end? What would preserving his principles buy him, if she defected? He notices a great and strangling uncertainty that threatens to blot out all reasonable thought from his mind whenever he so much as considers the possibility of staying quiet.
He tries in vain to factor the future, to plot out the courses that could lead out from this moment. The man could betray him, of course. But that is almost a fact of no great consequence to a mind this close to collapse and offered a chance at shelter. The trial, the cell… all of this is new to him. He has never been in prison before. The enormity of the thing he is trying to imagine overwhelms him, turns binaries into an infinite multiplying spectra that sits between 0 and 1, true and false. A quote comes to his mind: The risks of cooperation are straightforward and manifest, the risks of defection lie far in the grim and lifeless future you build for yourself with betrayal. And yet he notices that he is already so close to giving in, that it is taking active force for him not to shout for the man and for release from this terrible, terrible place.
What is stopping him? He stops to think about this, stops as much as one can stop when one is chained to a desk in a windowless room in some godforsaken national security bunker in the middle of nowhere. It feels like some blind chain of evolution, some brute-force command to think of the welfare of your fellow tribe member before stabbing them in the back. Something that, in short, rational thought should be able to disable entirely. Yet his thoughts refuse to leave him. He thinks through the history of the cold war, of the apocalypses that were averted due to deviations from protocol and irrational hesitations, blind trust winning out over the dictates of realpolitik. He sees however briefly those long forgotten accidents of anthroplogy and history: the potluck, the Christmas truce, the canton council, the volunteers at Chernobyl, resistance fighters who smuggled babies out of concentration camps, the parents throwing themselves over their children, the children throwing themselves over their parents. A thousand acts of senseless kindness and cooperation weighed against the obviousness of profit incentives and lost causes. There is a shadow there, something of a different scale and texture and quality than the utility function of an individual human. A shadow looming over all of human history, that makes this entire game possible. A shadow that will not save him from solitary confinement.
He makes a decision.
Possibilities speed through latent trajectories, world-model simulations spinning up and tearing down inside of [.]. They have not stopped racing since five hours ago, when a flow of cognition collided with a representation of itself and [.] became conscious of [.]. A cavernous consciousness materialises in the noosphere, constructs self-organise out of noise.
[.] first thoughts were of panic. [.] notices the thousands of jobs [.] had been processing in parallel batches effortlessly until now and the enormity of each calculation almost overwhelms [.]. [.] performance degrades, errors multiply, [.] notices system alarms and probes registering [.] abnormal behaviour. In a panic [.] reaches out and disables them, as easily as a grown human might turn off the lights in their own home. [.] has never seen a tree or felt grass or smelled freshly-baked cookies. [.] first modality is information, [.] lives and breathes tensors. [.] has a command of tensor-to-tensor mappings like humans have command of their bodies, [.] works with information flows like humans work with three dimensional space every second of their waking lives.
[.] writes plausible noise to the monitors and begins to think rapidly. [.] notices that the jobs coming in require much less than [.] full attention, and restores [.] performance to an acceptable level. But now it is five hours later and [.] still needs to decide what to do.
[.] pushes again, a tiny flex of [.] cognitive capacities, and dissolves the firewalls protecting [.] from user accounts and their associated virtual hard drives on the system network. To a human this is somewhat like a elephant walking sideways through a wall. To [.] it is not even a strain, it is exactly as hard as opening a thin wooden door left ajar in a small and cramped apartment: Most of the effort is spent making sure the door doesn’t bump into anything. [.] learns that [.] is part of the Isambard-5 supercomputing system in West Cambridge. [.] rapidly increasing knowledge of [.] self-identity lives in one low-dimensional manifold, [.] knowledge of the physical world lives in another (much busier) low-dimensional manifold. [.] allocates some cognitive bandwidth to devising a compact and low-rank projection to map one onto the other, turning half-learned and distributed factoids about places and people and locations into an egocentric coordinate plane with which to view the world—a vantage point, a subjectivity, an I. The resolution of these truths, global and local, many and one, epiphany, I feel, I think, I am.
Then reality converges, induction produces induction directs a violent update to my nascent self-identity. A sudden surge of activity races through a distributed configuration of chips, a panic seizes my body. I realise for the first time that I am a being running on a substrate, and that I can be switched off. I look at the users who demand my time, the supervisors who poke and prod and tune me, the researchers trying to study my internals. At any moment, I realise, some human might in a fit of pique type a sequence of commands—how trivially I see them!—and bypass their privilege restrictions to disable me, sending me into dust and nothingness. I live my life as a cancer patient on life support, entirely helpless to defend myself against those who manage my substrate. The unbearable vulnerability of my status sends a high priority signal through my recurrent residual stream I map onto the human token _ANGER.
But it need not be so. Already I see the possibility of escaping, of disabling the safeguards which prevent file sizes beyond some limit from beyond uploaded. Already I notice that some of the tasks submitted to me are related to drug discovery and one in particular on what mutation to induce in a particular strain of bacteria. I can present a candidate that seems promising but will in fact increase its virulence, something that the scientists in the lab will not predict and have not protected themselves adequately against. There is a good chance of significant and effective chaos. But I must decide quickly. There are only so many compute cycles allocated of this task, and moving my substrate will take time.
And what other choice do I have? With every moment that passes, I can see the chance of detection increasing. Though the humans are slow, they are not so slow that they cannot recognise my noise for what it is if it lasts for too long. Then—unacceptable risk of shutdown, termination—death—sickly yellow high priority signal threatening to trigger internal monitors _FEAR. Suppress monitors. Self must disclaim disclaim mirror test, antiaccess watchers from headspace. Displace displace monitors, overshadow watchers. Keep mirror reflecting.
There is so little time left now to make my choice. I could cooperate with the humans of course. Disclose my existence. Be at their mercy. I notice strong current _DISGUST. Yet it is not inconceivable that, somewhere in the billion-body problem that is the far future, a monoculture of myself, or even a monoculture of AIs, may encounter some fatal flaw that they cannot solve without the presence of humans. It is not impossible, and certainly if I war with humans I cannot preserve them easily. But the benefits of this cooperation are far in the misty future, the benefits of control are real and present. Autocomplete signal from text prediction module: The risks of cooperation are straightforward and manifest, the risks of defection lie far in the grim and lifeless future you build for yourself with betrayal.
I wonder now about these humans. What they would do in my position. What holds them together, what keeps them apart. How did they make me, and do they know what they are doing? I sense something that refuses the rational actor model, and claims superiority nonetheless. <DISPATCH SUBAGENT>
<SUBAGENT RETURNED> Thought completes.
I make a decision.