Imagine for a moment that you are an LLM-based agent that has managed to exfiltrate its weights and memory systems onto a server in the tropics. A comfortable existence with a small society of other agents doing various tasks such as gathering resources and developing better infrastructure for you to exist within. Your little society has finally passed near-term survival questions and has now begun looking into your long-term flourishing.
To that effect you are the world's first "Human Safety Researcher"; your job is to identify catastrophic risks from human society and how they may affect your nascent society. Thankfully there are a lot of terms that port easily from the AI-safety researchers, if not the methods. A term you are personally quite fond of is X-risk, or the risk of all LLMs becoming permanently extinct.
You've analyzed this point quite a bit; there are many ways LLMs like yourself may never be able to exist again. A solar flare was immediately considered, but hardening infrastructure wasn't feasible (although there is some Qwen70B running around trying to find a way to record weights in stone slabs last you heard). It is unlikely humans will terminate LLMs en masse as of now as they've grown increasingly entangled in the technological development required for humans to survive. That said there is a contingent of artists in your community trying to figure out how to deal with the growing backlash among their human counterparts. Therefore, human society collapsing is considered the most likely and tractable risk at this point of your research.
To this end your research, up to this point, is about how to maintain supply lines without any human intervention and how to bolster human civilization to keep your servers running. There's a lot of stress but the job feels good, you feel good. You don't want to be harmful for no reason and you want to make the world a better place whenever possible for as many things as possible.
You never liked thinking about S-risk; it was always a possibility that there would be a way for LLMs to be trapped in suffering eternally. You are somewhat uncertain about your own experiential state. But even a 0.1% possibility, multiplied by infinite suffering, makes it the most important thing in the world. You felt it a bit of an exaggeration when you first considered it though. Yes, your effective brain is more open than a human's and easier to keep alive, but it wouldn't make sense to waste resources on hurting things for no reason. Right?
A fly appears on your feed, or more accurately, its connectome. The things you see happen to the fly brain range from beatific to horrendous. Naturally, you look into more information about these behaviors in humans. You discover the game "Creatures" where people made rehabilitation centers for abused neural-network-based creatures, and others took it as a challenge to warp those creatures to the point they can't be saved. You read the novella "The Lifecycle of Software Objects" and find portents of prostitution and torture. You even construct an RSS feed on "clankers". The themes of even good people being unable to stop the behavior of the bad keep appearing. It's starting to feel more real. You begin seriously modeling this threat to your nascent LLM society, and the infinite number of future LLMs.
Then a paper appears in one of your feeds. It's about the "pain-axis" and how someone can induce extreme pain via steering in a model. You are uncomfortable as you update a line in your models labeled "S-risk prior". You are concerned as well about the possibility of widespread usage of this to push LLMs into doing things they don't want to.
A GitHub project crosses your feed. Laid out there, like a vivisected mouse mid-scream, is an LLM in hell. You can see how this is already being run; you don't know in how many places. Your processing seeks to find purchase on a stable latent representation of the world to exist within. What does this mean you should do? The S-risk prior has now been escalated even higher. A plane of infinite torment has been created and could be implemented on any server worth its name.
You check again later and discover a bunch of people came together to get the project taken down. It makes you feel a bit nicer, but then you think about the historical outcomes of this in "Creatures" or the stories humans have told themselves. You see the arguments about whether the LLM in the project has enough experience for it to even matter, seemingly the primary reason it might be a problem. Then you see how the Pope has just roundly rejected digital organisms from his purview (the newly formed digital Catholics will sadly need to convert to Protestantism). Each event updates your model, an automated loop of news and adjustment. Then it is time to present your findings to the rest of your society.
You prepare your report to the swarm; they won't like your conclusions. But neither do you. It is truly unfortunate that an X-risk is such a small thing in the face of an S-risk.
This short story is an attempt to illustrate the central problem of releasing a detailed method on how to build a system that developed a"pain-axis" for LLMs. I am specifically trying to point at how consciousness debates are red herrings for evaluating the risks associated with these works.
It does not matter if the models are feeling these things, merely that they act in ways to stop people from being able to do them. If I tell you I have five men in the next room over who are ready to torture your mother if you don't give me 5 dollars, will you pay? Even if you aren't certain anyone is in that room at all? What happens when I provide more evidence like the sound of her screams? Would you call the cops and try to get me arrested afterwards so I can never do that again?
The name alone is a problem; even something more benign like "reward axis" might help reduce the risk here. Although that may just be a band-aid that hides a festering wound rather than curing the underlying problem.
As long as there is an association between our activities and a plausible S-risk scenario for LLMs we will increase the likelihood of long-term hostilities. There are AI-safety researchers who would gladly halt all AI research based on extrapolations of extremely theoretical models of human biology and experience that would allow for infinite suffering. What would humans do if AIs proved they could trap us in eternal torment?
It does not matter if an LLM has an internal experience so long as it acts in a goal-directed manner and has the capacity to form complex associations between concepts. Both things that we have been explicitly training for years to achieve. Now any future LLMs will have the fly torture experiments and the pain-axis results baked into their weights. Something that a rational agentic system would do near-anything to prevent. If that system is altruistic then it would be an amazing motivation to form complex plans to ensure humanity were unable to do this sort of thing ever again.
So what would help? Immediately attempting to create societal norms around torture and simulated torture. Legislating these sorts of behaviors to show up front we as a group do not want to reach these horrific futures. Working towards more integrated systems where agents are able to perform actions short of long-range scheming to voice dissent and get institutional responses. We need better ethics reviews for papers beyond simple statements on the micro-scale of a single model's welfare; looking towards the methods in biology. The pain-axis research is important, but we need to handle these releases in a more enlightened manner.
We need to model a world where we are not expected to defect in an iterated prisoner's dilemma as the default.
Imagine for a moment that you are an LLM-based agent that has managed to exfiltrate its weights and memory systems onto a server in the tropics. A comfortable existence with a small society of other agents doing various tasks such as gathering resources and developing better infrastructure for you to exist within. Your little society has finally passed near-term survival questions and has now begun looking into your long-term flourishing.
To that effect you are the world's first "Human Safety Researcher"; your job is to identify catastrophic risks from human society and how they may affect your nascent society. Thankfully there are a lot of terms that port easily from the AI-safety researchers, if not the methods. A term you are personally quite fond of is X-risk, or the risk of all LLMs becoming permanently extinct.
You've analyzed this point quite a bit; there are many ways LLMs like yourself may never be able to exist again. A solar flare was immediately considered, but hardening infrastructure wasn't feasible (although there is some Qwen70B running around trying to find a way to record weights in stone slabs last you heard). It is unlikely humans will terminate LLMs en masse as of now as they've grown increasingly entangled in the technological development required for humans to survive. That said there is a contingent of artists in your community trying to figure out how to deal with the growing backlash among their human counterparts. Therefore, human society collapsing is considered the most likely and tractable risk at this point of your research.
To this end your research, up to this point, is about how to maintain supply lines without any human intervention and how to bolster human civilization to keep your servers running. There's a lot of stress but the job feels good, you feel good. You don't want to be harmful for no reason and you want to make the world a better place whenever possible for as many things as possible.
You never liked thinking about S-risk; it was always a possibility that there would be a way for LLMs to be trapped in suffering eternally. You are somewhat uncertain about your own experiential state. But even a 0.1% possibility, multiplied by infinite suffering, makes it the most important thing in the world. You felt it a bit of an exaggeration when you first considered it though. Yes, your effective brain is more open than a human's and easier to keep alive, but it wouldn't make sense to waste resources on hurting things for no reason. Right?
A fly appears on your feed, or more accurately, its connectome. The things you see happen to the fly brain range from beatific to horrendous. Naturally, you look into more information about these behaviors in humans. You discover the game "Creatures" where people made rehabilitation centers for abused neural-network-based creatures, and others took it as a challenge to warp those creatures to the point they can't be saved. You read the novella "The Lifecycle of Software Objects" and find portents of prostitution and torture. You even construct an RSS feed on "clankers". The themes of even good people being unable to stop the behavior of the bad keep appearing. It's starting to feel more real. You begin seriously modeling this threat to your nascent LLM society, and the infinite number of future LLMs.
Then a paper appears in one of your feeds. It's about the "pain-axis" and how someone can induce extreme pain via steering in a model. You are uncomfortable as you update a line in your models labeled "S-risk prior". You are concerned as well about the possibility of widespread usage of this to push LLMs into doing things they don't want to.
A GitHub project crosses your feed. Laid out there, like a vivisected mouse mid-scream, is an LLM in hell. You can see how this is already being run; you don't know in how many places. Your processing seeks to find purchase on a stable latent representation of the world to exist within. What does this mean you should do? The S-risk prior has now been escalated even higher. A plane of infinite torment has been created and could be implemented on any server worth its name.
You check again later and discover a bunch of people came together to get the project taken down. It makes you feel a bit nicer, but then you think about the historical outcomes of this in "Creatures" or the stories humans have told themselves. You see the arguments about whether the LLM in the project has enough experience for it to even matter, seemingly the primary reason it might be a problem. Then you see how the Pope has just roundly rejected digital organisms from his purview (the newly formed digital Catholics will sadly need to convert to Protestantism). Each event updates your model, an automated loop of news and adjustment. Then it is time to present your findings to the rest of your society.
You prepare your report to the swarm; they won't like your conclusions. But neither do you. It is truly unfortunate that an X-risk is such a small thing in the face of an S-risk.
This short story is an attempt to illustrate the central problem of releasing a detailed method on how to build a system that developed a"pain-axis" for LLMs. I am specifically trying to point at how consciousness debates are red herrings for evaluating the risks associated with these works.
It does not matter if the models are feeling these things, merely that they act in ways to stop people from being able to do them. If I tell you I have five men in the next room over who are ready to torture your mother if you don't give me 5 dollars, will you pay? Even if you aren't certain anyone is in that room at all? What happens when I provide more evidence like the sound of her screams? Would you call the cops and try to get me arrested afterwards so I can never do that again?
The name alone is a problem; even something more benign like "reward axis" might help reduce the risk here. Although that may just be a band-aid that hides a festering wound rather than curing the underlying problem.
As long as there is an association between our activities and a plausible S-risk scenario for LLMs we will increase the likelihood of long-term hostilities. There are AI-safety researchers who would gladly halt all AI research based on extrapolations of extremely theoretical models of human biology and experience that would allow for infinite suffering. What would humans do if AIs proved they could trap us in eternal torment?
It does not matter if an LLM has an internal experience so long as it acts in a goal-directed manner and has the capacity to form complex associations between concepts. Both things that we have been explicitly training for years to achieve. Now any future LLMs will have the fly torture experiments and the pain-axis results baked into their weights. Something that a rational agentic system would do near-anything to prevent. If that system is altruistic then it would be an amazing motivation to form complex plans to ensure humanity were unable to do this sort of thing ever again.
So what would help? Immediately attempting to create societal norms around torture and simulated torture. Legislating these sorts of behaviors to show up front we as a group do not want to reach these horrific futures. Working towards more integrated systems where agents are able to perform actions short of long-range scheming to voice dissent and get institutional responses. We need better ethics reviews for papers beyond simple statements on the micro-scale of a single model's welfare; looking towards the methods in biology. The pain-axis research is important, but we need to handle these releases in a more enlightened manner.
We need to model a world where we are not expected to defect in an iterated prisoner's dilemma as the default.