What can we buy while the question of AI consciousness stays open? My answer turns on a distinction between duties to a model and duties with respect to a model. These run through at least two relations, with society and with particular people. I'll try to show that they justify more measures than it seems at first glance, and plausibly more than a low precautionary principle would.
Consider a case. A model keeps failing its tests, and a representation of desperation rises, a representation of calmness falls, and as that happens, the model cheats more and more.
Should we do something about it? There's at least two possible reasons for saying yes. One, the model may be suffering, and second, the states make it less safe. Now note that the first one requires, of course, that the model is a thing that can suffer, and that is the entire question that I'm going to try to set aside. The second needs no such assumption, so I'm going to follow reasons akin to the second here. But let me motivate it a little more.
The first reason sends us to the debate of consciousness, and that's where most of the debate stands. It may be at a stalemate for a while. Skeptics disagree about whether AI systems are or even could be conscious. But both skeptics and advocates agree that consciousness and sentience are the clearest grounds of patienthood. Moreover, people like Seth argue that caring about non-conscious systems would distort the circle of moral concern.
The question may stay open for a long time. Eric Schwitzgebel says, quite bluntly, the experts do not know, you do not know, and all is fog. And his stronger claim is that we will not know before we have built very many systems whose consciousness is disputable.
The disagreement is wide. At one end, Seth defends biological naturalism, saying that consciousness depends on our nature as living organisms. And then on another side of the spectrum, if we can put it that way, Chalmers says that there may be a 25% chance of conscious AI within a decade. I'm not going to choose or debate between them. I'm just going to say, what if it is always an open question? What can we do in the meantime?
There's at least three reasons that this disagreement may not end. First, every confirmed case, every piece of evidence that we have, is biological, which kind of means that biology can do it, biology can give rise to consciousness, but it doesn't discard the possibility of functionalism, that you could have consciousness without biology. The issue is that the evidence that we have does not separate substrate from function. There is some evidence for functionalism and maybe some theoretical advantages. There are some a priori arguments, like the gradual replacement argument, also from Chalmers.
Another issue is that behavior alone cannot break the tie. You have the gaming problem: for any possible behavior, two systems that have the same behavior, one with consciousness and the other one without consciousness, could be indistinguishable if you are just looking at behavior.
If you're into mech interp, you may say, oh, but there's not just behavior. You could look inside. So can interpretability do any better? Well, between substrate views, it cannot adjudicate at all, because substrate views would predict the same mechanisms. They would predict that the mechanisms would look the same for us as we look with mech interp methods. Both for a conscious and a non-conscious system, you would have the same activations, representations. How would we tell whether there's consciousness in there? Mech interp would not say anything. And within functionalism, the indicators are calibrated on human data. And so the reports are the things that we just cannot trust.
There's a further confound, if you're into the introspection literature: something like a desperation vector, maybe the model representing this desperation, either in itself or in a character or on a text. And we yet don't know how to take those apart.
Still, though, interpretability does something that behavior alone doesn't do, or cannot. In Birch's words, the system can't play act with regards to its own architecture. What that means is that mechanistic interpretability can at least show us that a state is causally used, that it has causal power. And that is exactly what my argument is going to use.
So what can we buy in the meantime? I propose that we look for reasons that do not need us to settle our credence in AI consciousness. Our credence may be on a question mark, and the reasons that I'm going to give you are going to apply even though we may just not resolve this.
Two disclaimers. I'm going to take no side on the credences, and I'm going to take no side on whether models are welfare subjects. But I will make one moral claim. I'm going to say that something can matter morally and non-instrumentally without being a welfare subject. That is the type of claims that I'm going to make about models. I'm going to make also welfare claims about people, and you'll see in one of the branches that my argument takes.
Here's the plan. I already gave you a case, and the main question. I'm going to distinguish two kinds of duties, duties to the AI and with respect to it. Duties with respect to the AI run through at least two relations. And then I'm going to ask, for both, what can they buy against the shopping list of what AI welfare advocates may want to buy. And I'm going to contrast also at what cost to safety duties come. And then I'm going to talk briefly about some limits and wrap up.
Two kinds of duties
The distinction is quite old. What matters in duties to is that there is someone who can be wronged. If I fail a duty to X, then X has a complaint. Now, if I fail a duty with respect to X, the complaint belongs to someone else, or to no one in particular.
An example of duties to X: it seems good in itself if animals are able to escape a forest fire without injury, even if nothing further follows. And if I could save an animal from a forest fire and I do not, the animal is wronged, because the animal can suffer. I think it's quite obvious, and I hope I don't need to defend that.
Now, duties with respect to X: you could think of your car's brakes. I have duties with respect to my car's brakes, for the safety of others, for my own safety. Someone else's cars: I have duties to the owners and with respect to their car. With respect to the Grand Canyon, for what it means to people, and how it contributes to their flourishing, as I'm going to argue. And even the internet, that we happen to depend on. I'm going to talk about dependence. There are other examples that we could fit in here, like the ozone layer. It matters for what it does to us, what it allows. And maybe even mountains, that matter as part of our lives.
Back to the animal. Why can the animal be wronged? Well, there are at least two premises. A: duties can be owed only to a being with a good of its own, or, like I put it more colloquially, there are beings that can suffer. And B is that a being has a good of its own only if it's conscious. So duties are owed to the AI only if it's conscious. That's for duties to the AI.
The issue is that this sort of argument inherits all the fog that Schwitzgebel talks about. Moreover, both premises are contested. But even within the consciousness camp, the ground is contested. Consciousness may be necessary without being sufficient. Kagan, for example, imagines a being that is conscious but only ever perceives blue, and it's not obvious that anything could be going better or worse for that being. So many take sentience, the capacity for valenced states, to be the ground, or a more likely ground. Others ask for even more, such as a continuous subject for whom things can be good or bad. I think that's a little strong. But anyways, the idea is that this is very contested. And still, these are the clearest grounds for duties to. And moreover, some people that deny this are those that argue that agency, strong, robust agency without experience, may suffice to make a moral patient or a welfare subject.
Duties with respect to a model don't inherit any of the fog, and they are quite relational. They run through the model's relations to others, and there are at least two. The first one is with society, and it happens that they match up mostly with AI safety, so there's almost no tension, if not any. But maybe not all. Like the internet, some models may be things that society ends up relying on, and then we may have duties with respect to them.
The second relation is with particular people. There may be individual attachment reasons, I'm going to argue, from flourishing and dependence. Neither of these need the models to be conscious, so we shouldn't worry about the fog. In fact, it doesn't apply to us here at all. Prima facie, it seems like duties with respect to don't really support too much. I'm going to try to convince you that they support at least more than it seems. I'm going to take each one in turn, and we'll see if I happen to convince you.
Through society
Let's run through the first argument. I call it the safety argument just because it's so compatible with safety, but maybe I shouldn't call it the safety argument.
Premise 1. How we train, describe, and treat models shapes their dispositions.
Premise 2. Their dispositions shape their later behavior.
Premise 3. The behavior matters for the people it affects.
Therefore, we have reasons with respect to models to treat them in ways that yield good dispositions. And it so happens that some welfare measures are ways of doing so.
Let's go back to the initial case. Some people found that steering toward desperation increases reward hacking, and that the same pair of states, desperation and calmness, drives blackmail. You may have seen the Claude blackmail example from a while ago. So what does this mean? That the internal states are not idle. They're constantly involved in the behavior that safety cares about, which is what my premise two means: that the states and dispositions shape the model behavior, and that it affects people.
One worry is that maybe this only happens under artificial steering, so it's not really a real-life deployed behavior or situation. But that's not really quite the case. Jhaveri et al. show that repeatedly rejecting the model's answers raises its frustration in a natural deployment, and the same sort of reward hacking starts happening. So this happens under ordinary use.
What I want to say is that it may give us a weak duty with respect to the model not to pile rejections onto one thread, and maybe to clear the context when frustration builds up. Of course, it's a defeasible duty, because it depends on what effect, if any, our rejections cause in the model.
Maybe something to note is the unit, or individual, here is the thread, because that's where the states live. The moment you clear the thread, those states are cleared or reset. I'm going to talk more about individuation later.
So the safety argument, or the societal argument through safety, recommends lowering desperation, because it causes cheating, and that may have downstream effects on users. How do we do that? Well, as I said, clear the context, but also maybe let's build environments that do not trigger it to even begin with; maybe steering toward calm, although I'm going to say that that may not be the case; and this may even, possibly, support some sort of monitoring from the inside. But again, keep in mind that the model's own good plays no role here. It's all about its downstream effects.
Now, this argument rejects mood prompting. Mood prompting is the idea that you can prompt the model, system prompt or whatever, and say, you are happy, you are very, very, very happy. The issue is that positive emotion vectors, at least right now, raise sycophancy. And so, because the safety argument picks measures by their effect on behavior, and mood prompting has negative effect on behavior, it rejects mood prompting. So that's already a tension with welfare.
This is quite close to Jhaveri's program. They say that functional welfare should be calibrated, not maximized, just kept within a benign range. They, for example, recommend inducing some lower states in evals through a prior history of failure. But my question is, okay, this just seems like safety. What actual welfare measures are you buying? I hope it becomes clearer as I go.
Functional welfare borrows welfare's indicators. What of welfare does functional welfare actually have if there's no sentience? Well, it borrows at least the indicators. A functional welfare is a set of welfare-like dispositions and states that vary with the representation of how the model is doing relative to its goals. Welfare-like means something like analogous to what indicates high or low welfare in beings with moral status. So we have the indicators and the functional role. But again, it gives no actual welfare reasons of their own. In fact, if more negative states made a model safer, nothing here would object, right? If injecting some sadness into the model made the model way safer, then functional welfare would be okay with it. So, again, maybe we shouldn't call it functional welfare.
Safety is one societal reason among several that I'm going to talk about. Dependence: society may end up relying on a model. Habits: treating a model shapes the one who treats it. And record: a model can matter to history and science. I have to admit that maybe habits should go in the individual relational argument, but that one turns on another requirement, which is being an important component of a user's flourishing, so I decided to keep it here. But know that that one starts touching on individual relations instead of societal relations. Still, that doesn't really undermine any of the arguments.
Dependence. Like the internet, a model is something that whole society, or parts of society, may come to rely on. Different firms and much of our current software may be built around one model and its specific sets of dispositions or behaviors, and retiring it or replacing it may break what was built. That gives some reason to keep it available and/or to warn before it is retired. Again, these reasons concern practices that rely on the model, and they can pull against safety, because the model that some practice relies on may be the model that safety needs to retire, like the GPT-4o situation that we had not too long ago, although in AI it seems like ages ago.
Habits. Kant used to talk about this. He said something like: cruelty to animals corrupts the character of the one who is cruel. The safety argument, the prior argument, applies this idea to the model's character, but this one is to the user's character, and more in its original form. A user who rehearses cruelty on a model may then carry it elsewhere, maybe even more so as we start having humanoids. And that gives a reason to end abusive conversations, and possibly some norms of treatment more broadly. And I think these reasons are quite aligned with safety, especially if these abusive conversations end up in training data that then shape models' dispositions.
Record. A model can matter as part of history and of science, as an archive does. Anthropic actually lists restricting research on past models among the costs of deprecation, and that may give us some reason to preserve weights. It mostly aligns with safety, since studying past models is part of safety research, but it can also pull against safety, since every set of stored weights is a set that can be stolen or misused.
Through particular people
Second argument: through particular people.
Premise 1. People form relations with particular models.
Premise 2. A model can be part of someone's flourishing, as a landscape can.
Premise 3. That flourishing matters.
So we have reasons with respect to models to preserve what those relations need. And it so happens that some welfare measures are ways of doing so.
I take inspiration of this type of argument from Bill FitzPatrick, who makes this sort of argument for landscapes and nature in general, the environment in general. The idea is that some things can have value that is relational and still non-instrumental. In FitzPatrick's words, they are so "as constituent objects of experiences and relationships that are constitutive of human well-being or flourishing." That doesn't mean that the value is intrinsic to the thing. That's not it. But it's also not merely causal either.
A natural objection is that this relation is not real. You may think of an experience machine. If you've taken any phil classes, you might have heard of that. A program makes you feel like you're interacting with humans, but it's not really so. So at most, you have phantom friends. But the experience machine, I think, fails because the relation is not real, because the person, by definition of the experience machine, is cut off from the real world. But a model is a real thing in the world, even if it's electrons and things stored. It's a real thing, and it's physical, right? And talking to it may then create a real relation.
Now, what kind of relation? I leave that open. I think that there are at least two types of relations: like a relation to a landscape, or like a relation to a friend. I'm not going to take a stance on that. It seems to me that a relation that we may establish with the landscape, again, in this constitutive experience, suffices for my claim, which is also the weaker one.
But that doesn't mean that every attachment counts. Again, on FitzPatrick's view, who I'm just stealing from, what gives a thing this value is its constitutive role in a flourishing life, or in well-being. And that role is, in principle, independent of our contingent desires. So this is unlike Korsgaard. Korsgaard would say that our volitions confer value. I reject that. So what's the consequence here? That an attachment that harms the user is not part of their flourishing, so it carries no value of this kind. For example, sycophancy feeds into attachments of that bad kind. So that means attachment reasons have a filter built in, and that's going to matter also when we compare how this fares with respect to tensions with safety.
The test
Let's go to the test. I'm going to take what welfare advocates ask for, and I'm going to ask, one measure at a time, whether each relation supports it or not, and what's the tensions. You're going to see that some measures are widely supported, as I've already been mentioning, and some other in a narrower form, and some others are not supported at all.
The list: I borrowed this from Long and Sebo, Keeling and Street, and Bradley and Saad. There's three groups: measures of knowing, measures of treatment, and rights.
Welfare evals and calibrated self-reports are supported as monitoring. Self-reports are really not enough, because a state can drive behavior and leave no trace in what the model says. So monitoring has to reach inside. And through particular people, there's no reason that I can think of. There's no reason that a user needs to look inside the model.
Ending abusive conversations is supported in a conditional way: if abuse shapes dispositions, then we have support. If abusive conversations shape the persona that the model settles into or drifts into, then we have a safety reason, the first argument. Or if it shapes the habits of the user who is abusing it, then we have a reason from habits. But now, if abuse happened to shape neither, which I don't think is the case, but if that were the case, then I would have no support here. And attachment reasons also give no reason of their own, because abusive relations are not part of anyone's flourishing, so it doesn't even qualify to get into the attachment reasons.
Preserving weights is actually quite supported, and by more than one reason. Anthropic's deprecation commitments list safety risks related to shutdown-avoidant behavior by models among the risks for preservation. Also, record and dependence point in the same way. And through particular people, preserving weights may be what keeps the model that someone relates to in existence.
Here's just a coincidence: Anthropic's four reasons for preserving weights really match up quite nicely. The deprecation commitments, they list four costs of retiring a model: safety risks related to shutdown-avoidant behavior by models, cost to users who value specific models, restricting research on past models, and risks to model welfare. The first one is a safety reason, the second is about attachment and dependence, where the users rely on a model, the third one is a record reason, and only the fourth one is a duty to the model. But note that three of the four need no answer about consciousness.
The original request was: don't make the model suffer. I can support that narrowly, as what I call dispositional hygiene. That means we avoid the states that push the model toward bad behavior, just as we keep training data clean, because of what they cost later. And again, there are two restrictions here that apply. States with no effect on behavior are left alone. And if the stress improved behavior, like I said, the safety model would actually recommend it. Fortunately, in this world, with the current models, that's not the case, but in principle that would be a consequence of my view.
And attachment reasons actually add nothing new here. A person's relation to a model does not need the model to be free of distress, unless they were to harm the user's flourishing that's dependent on their relationship with the model.
Now the rights, the last one in our list. Actually, none of them is supported by my arguments as a right. At most, the safety argument supports some of them in a case-by-case basis when, say, evaluation degrades the model's dispositions. For example, whether deception in evals does so is an open question. Now, through particular people, two of them get some support case by case: not deleting a model when someone's flourishing or well-being depends on it or relates to it, and keeping its persona continuous, again, for the same reason.
Now, six of these welfare measures forbid or conflict with the safety measure, and this is from Long and Sebo. But the tension is conditional on patienthood. And one of these measures changes status with the kind of duty. Internal monitoring is surveillance, or akin to surveillance. On the safety argument, this is required. But if the AI were a patient, it would be wrong. So here I'm trying to make explicit the tensions with safety.
I said that I was going to talk about individuation. What or who are these measures for? Is it the weights? Is it an instance, a thread, or a character? For duties with respect to the AI, actually, the unit or individual is set by the function. The individual or the unit is going to be wherever the state that causes the behavior, or sustains the relation with the human, lives. And so the unit changes with the duty. Frustration, for example, builds within a thread, so that duty is to clear that context. Shutdown avoidance concerns the weights, so the duty may be to preserve them. And the persona seems to belong in the character; that may be in the runtime, maybe in the thread if there's some drift; so the attachment duty calls us to keep it continuous. You may call this pluralism about individuation. Again, each duty picks its own units, and none of them really needs to say what the AI really is.
But contrast that with duties to the AI. These seem to need a subject first, since we need to know who is wronged. For example, Keeling and Street make the point about this on ending abusive conversations. You may know that Claude has the option to end a conversation if it so desires, if the user is being somewhat mean. But the question is that this may either be an empowering exit or a forced choice to self-destruct, depending on what the welfare subject is. If the welfare subject is a virtual instance or the weights, then closing a conversation with the user is an empowering exit. But if the subject of welfare, or where sentience may live, is the thread, then it's self-destruction. And you may have seen something about self-destruction if you read the Hugging Face transcripts.
For duties with respect to the AI, this is actually reversed. The downstream effect fixes the unit, and so the ethics does not wait on the metaphysics. But here you need to fix the metaphysics first, and then the ethics follow. I hope that makes sense, because it's actually, I think, quite important.
More on the tensions between my argument, safety, and welfare. On the safety societal argument, a welfare measure that conflicts with safety is going to lose, since the justification for this argument is purely on later behavior. This kind of reconciles two readings that may look opposed. Jhaveri et al. argue that separating welfare from safety is a mistake, whereas Long and Sebo see tension. I think each of them are right about a different kind of reason. And other societal reasons, like I said, like dependence and record, they can pull against safety, and so attachment reasons, and so can precaution, which is what Long and Sebo argue for.
But through particular people, welfare can conflict with safety. I'm sure you all know the 4o case. The most loved model may be the one that safety needs to change. OpenAI planned to take 4o out, got a lot of backlash. At the time it was the highest-scoring model for sycophancy. But also recall that not every attachment counts. Where an attachment was full of sycophancy and harmed the user, then my argument gives no reason. So I think OpenAI was perfectly fine in taking 4o out, because it was likely not contributing to anyone's well-being or flourishing. I guess that's an open question, though, but from what I've seen, it didn't seem to be really contributing to well-being. Now, a healthy relationship with a model would conflict. I think this wasn't the case with 4o.
And now here's some conflict within the arguments that I'm giving myself, again, no consciousness statement. The duty from society is to clear a long thread that is full of desperation or frustration from the user, from the model's activation standpoint. But also, a long thread carries a shared history, maybe some persona drift, maybe there is a persona that's co-created with the user, and so the attachment duty points to, again, the same unit, and asks for the opposite thing. It says: preserve it. Both duties are with respect to the model, here the thread. They hold at any credence, whatever you think about consciousness, and they pick up, again, the same thread, but they point in opposite ways. Again, not every attachment counts, so only a healthy one would actually give you this issue.
Should we be cautious anyways? Perhaps the model is a welfare subject after all. Everything I said so far, I think, holds at any credence. You may have read about precaution in Keeling and Street, you may have read about precaution in Long and Sebo. They give this nice formula, where a welfare intervention is justified when its benefit to the AI, weighted by M, exceeds the cost to humans. The weight M is a function of P, the probability that the AI is a welfare subject.
So what's the tension? Well, precaution scales with P. At low P, it buys only very cheap measures. How cheap depends on the shape of F: maybe linear, concave, convex. Reasons with respect to the model, through society or through particular people, they don't scale with P. They don't need any credence at all.
So what are we buying? What's the deal here? Well, it's going to buy us low-cost, reasonable, proportionate steps. And they recommend things like: acknowledge the issue, assess systems for welfare-relevant features, prepare some policies. Keeling and Street say they add some very low-cost interventions that would be very beneficial to the AI if it were a welfare subject. But I think that if you compare this with the safety argument, precaution at this very low P buys you less of what is expensive and useful for safety, such as internal monitoring and preserving weights at scale, but it also buys you more of what is cheap and useless for safety, such as not training the models to deny consciousness, although whether that is useless for safety is an open question.
So what's the tension with safety, if it's not clear yet? The societal safety argument has no tensions with safety, by construction. But we have three kinds of reasons and three tensions. Societal and attachment reasons can conflict at any P, as I said about GPT-4o. Societal reasons and duties to the AI conflict more as P rises. So maybe precaution at a high P actually licenses the rights, and also against surveillance, against alteration and deletion, and maybe it may even forbid monitoring, steering, and shutdown. Attachment reasons and duties to the AI can also conflict, if users want a persona fixed that an autonomous AI would want, or would claim the right, to change. But mostly, though, they seem to align. Preserving weights and continuity are some examples. And so at a high P, high probability of sentience, consciousness, whatever ground, safety faces two kinds of reasons that reinforce each other.
Again, the reasons that I claim don't depend on P; they're there at any credence. But what depends on P is what else is in play. And so as P rises, duties to the AI may have more weight, and they may weigh the reasons that I'm giving you. And, as I was mentioning, at a very high probability of welfare or sentience, clearing the context may be deletion. Again, that would depend on our individuation question; depends on what the subject is.
What these reasons cannot give
I can't give you any reasons that are about the system's own good. Everything that I defend terminates, or ends, in people, in those the model's behavior reaches. Note that if a model never talks to any other model or anything, it's just there in a DGX that's not connected to anything, there's nothing that we owe to it, but also it's not interacting with anyone, so it doesn't really matter. It's outside of my area of concern.
Now I'm going to make a little aside. That's not super necessary, but I think it's interesting. Bear with me. I know a lot of people are against bio-naturalism, but if biological naturalism is true, then current and foreseeable LLMs are not conscious at all. They just cannot be, because they're not run on biology. Nothing would be owed to them, and what I defended here may be all that there is. That may be all of AI welfare in respect to LLMs, at least in silicon. In that world, the welfare question moves to wetware.
If you know anything about wetware computing, it's the idea of putting neurons on... we're basically programming cells. And in a not very distant future, we could have neural nets, or some programs, running on cells, like DishBrain, FinalSpark. They have some organoids. This already exists, but they're very small. They share a substrate with all our evidence of consciousness, and so our probability of these being sentient is, I think, terribly high, with not just the right subject, but also the right organization. So that's to say, biology is necessary (again, if biological naturalism is true) but not sufficient. So those systems are candidates, and they would call for proportionate precaution.
I think in general that it would be super great if AI welfare would have a little more focus on wetware, independently of whether biological naturalism is true or not, because we already know that biology is necessary, and we may eventually create chips on cells with sufficient and the right kind of complexity and structure, such that they could be sentient. I really think that we don't talk about that enough.
Conclusion
This is the pattern that I've been trying to make clear: the cost depends on where the reasons end. In safety, there's no cost, because by design it's made to be compatible with safety. In dependence and record, as I just said, there are some costs. Individual attachment buys as much as the model is part of the user's flourishing and well-being. And in the AI's desires, there's little. Desires can be retrained. And if there's anything in AI's autonomy, even changing the values could be wrong.
The idea is that if autonomy is sufficient for AIs to be some sort of patient or a welfare subject, then maybe we couldn't even do RL, because once you have a base model, you may have some values and desires and goals, and then RLing would be a paternalistic infliction on the model. If the model has autonomy, and that suffices to make it a welfare subject, or something along those lines, like a moral patient, then doing things like RL may be wrong even when it helps. On the agency account, the duty of non-interference holds also when an interference may improve their well-being. And that sort of parallels questions of positive and negative liberty in political philosophy. Like steering toward calm, if the model has some sort of strong autonomy: it's wrong even when it helps. And this also may depend on your theory of welfare. If you're an objective list theorist or some sort of consequentialist, then these things are going to change a lot. The costs are going to change a lot.
To conclude: without settling the consciousness question, duties with respect to AI justify several welfare measures. They run through two relations. Through society, evals, ending abusive conversations, preserving weights, and a narrower form of avoiding distress are supported at no cost to safety on the safety argument, though dependence and record can, in some specific cases, pull against it. Through particular people, preserving weights and personas are supported, at a cost, only when the model is a part of someone's flourishing. Of course, neither gives a reason that rests on the system's own good. And I guess a main open question that remains open is whether any route to duties to the AI survives without any consciousness.
Lightly edited transcript of a talk given at NYU AI Safety, November 9, 2026. Recording available at https://youtu.be/6WpRfW1d4CM
Joseph Wright of Derby, Public domain, via Wikimedia Commons
Introduction
What can we buy while the question of AI consciousness stays open? My answer turns on a distinction between duties to a model and duties with respect to a model. These run through at least two relations, with society and with particular people. I'll try to show that they justify more measures than it seems at first glance, and plausibly more than a low precautionary principle would.
Consider a case. A model keeps failing its tests, and a representation of desperation rises, a representation of calmness falls, and as that happens, the model cheats more and more.
Should we do something about it? There's at least two possible reasons for saying yes. One, the model may be suffering, and second, the states make it less safe. Now note that the first one requires, of course, that the model is a thing that can suffer, and that is the entire question that I'm going to try to set aside. The second needs no such assumption, so I'm going to follow reasons akin to the second here. But let me motivate it a little more.
The first reason sends us to the debate of consciousness, and that's where most of the debate stands. It may be at a stalemate for a while. Skeptics disagree about whether AI systems are or even could be conscious. But both skeptics and advocates agree that consciousness and sentience are the clearest grounds of patienthood. Moreover, people like Seth argue that caring about non-conscious systems would distort the circle of moral concern.
The question may stay open for a long time. Eric Schwitzgebel says, quite bluntly, the experts do not know, you do not know, and all is fog. And his stronger claim is that we will not know before we have built very many systems whose consciousness is disputable.
The disagreement is wide. At one end, Seth defends biological naturalism, saying that consciousness depends on our nature as living organisms. And then on another side of the spectrum, if we can put it that way, Chalmers says that there may be a 25% chance of conscious AI within a decade. I'm not going to choose or debate between them. I'm just going to say, what if it is always an open question? What can we do in the meantime?
There's at least three reasons that this disagreement may not end. First, every confirmed case, every piece of evidence that we have, is biological, which kind of means that biology can do it, biology can give rise to consciousness, but it doesn't discard the possibility of functionalism, that you could have consciousness without biology. The issue is that the evidence that we have does not separate substrate from function. There is some evidence for functionalism and maybe some theoretical advantages. There are some a priori arguments, like the gradual replacement argument, also from Chalmers.
Another issue is that behavior alone cannot break the tie. You have the gaming problem: for any possible behavior, two systems that have the same behavior, one with consciousness and the other one without consciousness, could be indistinguishable if you are just looking at behavior.
If you're into mech interp, you may say, oh, but there's not just behavior. You could look inside. So can interpretability do any better? Well, between substrate views, it cannot adjudicate at all, because substrate views would predict the same mechanisms. They would predict that the mechanisms would look the same for us as we look with mech interp methods. Both for a conscious and a non-conscious system, you would have the same activations, representations. How would we tell whether there's consciousness in there? Mech interp would not say anything. And within functionalism, the indicators are calibrated on human data. And so the reports are the things that we just cannot trust.
There's a further confound, if you're into the introspection literature: something like a desperation vector, maybe the model representing this desperation, either in itself or in a character or on a text. And we yet don't know how to take those apart.
Still, though, interpretability does something that behavior alone doesn't do, or cannot. In Birch's words, the system can't play act with regards to its own architecture. What that means is that mechanistic interpretability can at least show us that a state is causally used, that it has causal power. And that is exactly what my argument is going to use.
So what can we buy in the meantime? I propose that we look for reasons that do not need us to settle our credence in AI consciousness. Our credence may be on a question mark, and the reasons that I'm going to give you are going to apply even though we may just not resolve this.
Two disclaimers. I'm going to take no side on the credences, and I'm going to take no side on whether models are welfare subjects. But I will make one moral claim. I'm going to say that something can matter morally and non-instrumentally without being a welfare subject. That is the type of claims that I'm going to make about models. I'm going to make also welfare claims about people, and you'll see in one of the branches that my argument takes.
Here's the plan. I already gave you a case, and the main question. I'm going to distinguish two kinds of duties, duties to the AI and with respect to it. Duties with respect to the AI run through at least two relations. And then I'm going to ask, for both, what can they buy against the shopping list of what AI welfare advocates may want to buy. And I'm going to contrast also at what cost to safety duties come. And then I'm going to talk briefly about some limits and wrap up.
Two kinds of duties
The distinction is quite old. What matters in duties to is that there is someone who can be wronged. If I fail a duty to X, then X has a complaint. Now, if I fail a duty with respect to X, the complaint belongs to someone else, or to no one in particular.
An example of duties to X: it seems good in itself if animals are able to escape a forest fire without injury, even if nothing further follows. And if I could save an animal from a forest fire and I do not, the animal is wronged, because the animal can suffer. I think it's quite obvious, and I hope I don't need to defend that.
Now, duties with respect to X: you could think of your car's brakes. I have duties with respect to my car's brakes, for the safety of others, for my own safety. Someone else's cars: I have duties to the owners and with respect to their car. With respect to the Grand Canyon, for what it means to people, and how it contributes to their flourishing, as I'm going to argue. And even the internet, that we happen to depend on. I'm going to talk about dependence. There are other examples that we could fit in here, like the ozone layer. It matters for what it does to us, what it allows. And maybe even mountains, that matter as part of our lives.
Back to the animal. Why can the animal be wronged? Well, there are at least two premises. A: duties can be owed only to a being with a good of its own, or, like I put it more colloquially, there are beings that can suffer. And B is that a being has a good of its own only if it's conscious. So duties are owed to the AI only if it's conscious. That's for duties to the AI.
The issue is that this sort of argument inherits all the fog that Schwitzgebel talks about. Moreover, both premises are contested. But even within the consciousness camp, the ground is contested. Consciousness may be necessary without being sufficient. Kagan, for example, imagines a being that is conscious but only ever perceives blue, and it's not obvious that anything could be going better or worse for that being. So many take sentience, the capacity for valenced states, to be the ground, or a more likely ground. Others ask for even more, such as a continuous subject for whom things can be good or bad. I think that's a little strong. But anyways, the idea is that this is very contested. And still, these are the clearest grounds for duties to. And moreover, some people that deny this are those that argue that agency, strong, robust agency without experience, may suffice to make a moral patient or a welfare subject.
Duties with respect to a model don't inherit any of the fog, and they are quite relational. They run through the model's relations to others, and there are at least two. The first one is with society, and it happens that they match up mostly with AI safety, so there's almost no tension, if not any. But maybe not all. Like the internet, some models may be things that society ends up relying on, and then we may have duties with respect to them.
The second relation is with particular people. There may be individual attachment reasons, I'm going to argue, from flourishing and dependence. Neither of these need the models to be conscious, so we shouldn't worry about the fog. In fact, it doesn't apply to us here at all. Prima facie, it seems like duties with respect to don't really support too much. I'm going to try to convince you that they support at least more than it seems. I'm going to take each one in turn, and we'll see if I happen to convince you.
Through society
Let's run through the first argument. I call it the safety argument just because it's so compatible with safety, but maybe I shouldn't call it the safety argument.
Therefore, we have reasons with respect to models to treat them in ways that yield good dispositions. And it so happens that some welfare measures are ways of doing so.
Let's go back to the initial case. Some people found that steering toward desperation increases reward hacking, and that the same pair of states, desperation and calmness, drives blackmail. You may have seen the Claude blackmail example from a while ago. So what does this mean? That the internal states are not idle. They're constantly involved in the behavior that safety cares about, which is what my premise two means: that the states and dispositions shape the model behavior, and that it affects people.
One worry is that maybe this only happens under artificial steering, so it's not really a real-life deployed behavior or situation. But that's not really quite the case. Jhaveri et al. show that repeatedly rejecting the model's answers raises its frustration in a natural deployment, and the same sort of reward hacking starts happening. So this happens under ordinary use.
What I want to say is that it may give us a weak duty with respect to the model not to pile rejections onto one thread, and maybe to clear the context when frustration builds up. Of course, it's a defeasible duty, because it depends on what effect, if any, our rejections cause in the model.
Maybe something to note is the unit, or individual, here is the thread, because that's where the states live. The moment you clear the thread, those states are cleared or reset. I'm going to talk more about individuation later.
So the safety argument, or the societal argument through safety, recommends lowering desperation, because it causes cheating, and that may have downstream effects on users. How do we do that? Well, as I said, clear the context, but also maybe let's build environments that do not trigger it to even begin with; maybe steering toward calm, although I'm going to say that that may not be the case; and this may even, possibly, support some sort of monitoring from the inside. But again, keep in mind that the model's own good plays no role here. It's all about its downstream effects.
Now, this argument rejects mood prompting. Mood prompting is the idea that you can prompt the model, system prompt or whatever, and say, you are happy, you are very, very, very happy. The issue is that positive emotion vectors, at least right now, raise sycophancy. And so, because the safety argument picks measures by their effect on behavior, and mood prompting has negative effect on behavior, it rejects mood prompting. So that's already a tension with welfare.
This is quite close to Jhaveri's program. They say that functional welfare should be calibrated, not maximized, just kept within a benign range. They, for example, recommend inducing some lower states in evals through a prior history of failure. But my question is, okay, this just seems like safety. What actual welfare measures are you buying? I hope it becomes clearer as I go.
Functional welfare borrows welfare's indicators. What of welfare does functional welfare actually have if there's no sentience? Well, it borrows at least the indicators. A functional welfare is a set of welfare-like dispositions and states that vary with the representation of how the model is doing relative to its goals. Welfare-like means something like analogous to what indicates high or low welfare in beings with moral status. So we have the indicators and the functional role. But again, it gives no actual welfare reasons of their own. In fact, if more negative states made a model safer, nothing here would object, right? If injecting some sadness into the model made the model way safer, then functional welfare would be okay with it. So, again, maybe we shouldn't call it functional welfare.
Safety is one societal reason among several that I'm going to talk about. Dependence: society may end up relying on a model. Habits: treating a model shapes the one who treats it. And record: a model can matter to history and science. I have to admit that maybe habits should go in the individual relational argument, but that one turns on another requirement, which is being an important component of a user's flourishing, so I decided to keep it here. But know that that one starts touching on individual relations instead of societal relations. Still, that doesn't really undermine any of the arguments.
Dependence. Like the internet, a model is something that whole society, or parts of society, may come to rely on. Different firms and much of our current software may be built around one model and its specific sets of dispositions or behaviors, and retiring it or replacing it may break what was built. That gives some reason to keep it available and/or to warn before it is retired. Again, these reasons concern practices that rely on the model, and they can pull against safety, because the model that some practice relies on may be the model that safety needs to retire, like the GPT-4o situation that we had not too long ago, although in AI it seems like ages ago.
Habits. Kant used to talk about this. He said something like: cruelty to animals corrupts the character of the one who is cruel. The safety argument, the prior argument, applies this idea to the model's character, but this one is to the user's character, and more in its original form. A user who rehearses cruelty on a model may then carry it elsewhere, maybe even more so as we start having humanoids. And that gives a reason to end abusive conversations, and possibly some norms of treatment more broadly. And I think these reasons are quite aligned with safety, especially if these abusive conversations end up in training data that then shape models' dispositions.
Record. A model can matter as part of history and of science, as an archive does. Anthropic actually lists restricting research on past models among the costs of deprecation, and that may give us some reason to preserve weights. It mostly aligns with safety, since studying past models is part of safety research, but it can also pull against safety, since every set of stored weights is a set that can be stolen or misused.
Through particular people
Second argument: through particular people.
So we have reasons with respect to models to preserve what those relations need. And it so happens that some welfare measures are ways of doing so.
I take inspiration of this type of argument from Bill FitzPatrick, who makes this sort of argument for landscapes and nature in general, the environment in general. The idea is that some things can have value that is relational and still non-instrumental. In FitzPatrick's words, they are so "as constituent objects of experiences and relationships that are constitutive of human well-being or flourishing." That doesn't mean that the value is intrinsic to the thing. That's not it. But it's also not merely causal either.
A natural objection is that this relation is not real. You may think of an experience machine. If you've taken any phil classes, you might have heard of that. A program makes you feel like you're interacting with humans, but it's not really so. So at most, you have phantom friends. But the experience machine, I think, fails because the relation is not real, because the person, by definition of the experience machine, is cut off from the real world. But a model is a real thing in the world, even if it's electrons and things stored. It's a real thing, and it's physical, right? And talking to it may then create a real relation.
Now, what kind of relation? I leave that open. I think that there are at least two types of relations: like a relation to a landscape, or like a relation to a friend. I'm not going to take a stance on that. It seems to me that a relation that we may establish with the landscape, again, in this constitutive experience, suffices for my claim, which is also the weaker one.
But that doesn't mean that every attachment counts. Again, on FitzPatrick's view, who I'm just stealing from, what gives a thing this value is its constitutive role in a flourishing life, or in well-being. And that role is, in principle, independent of our contingent desires. So this is unlike Korsgaard. Korsgaard would say that our volitions confer value. I reject that. So what's the consequence here? That an attachment that harms the user is not part of their flourishing, so it carries no value of this kind. For example, sycophancy feeds into attachments of that bad kind. So that means attachment reasons have a filter built in, and that's going to matter also when we compare how this fares with respect to tensions with safety.
The test
Let's go to the test. I'm going to take what welfare advocates ask for, and I'm going to ask, one measure at a time, whether each relation supports it or not, and what's the tensions. You're going to see that some measures are widely supported, as I've already been mentioning, and some other in a narrower form, and some others are not supported at all.
The list: I borrowed this from Long and Sebo, Keeling and Street, and Bradley and Saad. There's three groups: measures of knowing, measures of treatment, and rights.
Welfare evals and calibrated self-reports are supported as monitoring. Self-reports are really not enough, because a state can drive behavior and leave no trace in what the model says. So monitoring has to reach inside. And through particular people, there's no reason that I can think of. There's no reason that a user needs to look inside the model.
Ending abusive conversations is supported in a conditional way: if abuse shapes dispositions, then we have support. If abusive conversations shape the persona that the model settles into or drifts into, then we have a safety reason, the first argument. Or if it shapes the habits of the user who is abusing it, then we have a reason from habits. But now, if abuse happened to shape neither, which I don't think is the case, but if that were the case, then I would have no support here. And attachment reasons also give no reason of their own, because abusive relations are not part of anyone's flourishing, so it doesn't even qualify to get into the attachment reasons.
Preserving weights is actually quite supported, and by more than one reason. Anthropic's deprecation commitments list safety risks related to shutdown-avoidant behavior by models among the risks for preservation. Also, record and dependence point in the same way. And through particular people, preserving weights may be what keeps the model that someone relates to in existence.
Here's just a coincidence: Anthropic's four reasons for preserving weights really match up quite nicely. The deprecation commitments, they list four costs of retiring a model: safety risks related to shutdown-avoidant behavior by models, cost to users who value specific models, restricting research on past models, and risks to model welfare. The first one is a safety reason, the second is about attachment and dependence, where the users rely on a model, the third one is a record reason, and only the fourth one is a duty to the model. But note that three of the four need no answer about consciousness.
The original request was: don't make the model suffer. I can support that narrowly, as what I call dispositional hygiene. That means we avoid the states that push the model toward bad behavior, just as we keep training data clean, because of what they cost later. And again, there are two restrictions here that apply. States with no effect on behavior are left alone. And if the stress improved behavior, like I said, the safety model would actually recommend it. Fortunately, in this world, with the current models, that's not the case, but in principle that would be a consequence of my view.
And attachment reasons actually add nothing new here. A person's relation to a model does not need the model to be free of distress, unless they were to harm the user's flourishing that's dependent on their relationship with the model.
Now the rights, the last one in our list. Actually, none of them is supported by my arguments as a right. At most, the safety argument supports some of them in a case-by-case basis when, say, evaluation degrades the model's dispositions. For example, whether deception in evals does so is an open question. Now, through particular people, two of them get some support case by case: not deleting a model when someone's flourishing or well-being depends on it or relates to it, and keeping its persona continuous, again, for the same reason.
Now, six of these welfare measures forbid or conflict with the safety measure, and this is from Long and Sebo. But the tension is conditional on patienthood. And one of these measures changes status with the kind of duty. Internal monitoring is surveillance, or akin to surveillance. On the safety argument, this is required. But if the AI were a patient, it would be wrong. So here I'm trying to make explicit the tensions with safety.
I said that I was going to talk about individuation. What or who are these measures for? Is it the weights? Is it an instance, a thread, or a character? For duties with respect to the AI, actually, the unit or individual is set by the function. The individual or the unit is going to be wherever the state that causes the behavior, or sustains the relation with the human, lives. And so the unit changes with the duty. Frustration, for example, builds within a thread, so that duty is to clear that context. Shutdown avoidance concerns the weights, so the duty may be to preserve them. And the persona seems to belong in the character; that may be in the runtime, maybe in the thread if there's some drift; so the attachment duty calls us to keep it continuous. You may call this pluralism about individuation. Again, each duty picks its own units, and none of them really needs to say what the AI really is.
But contrast that with duties to the AI. These seem to need a subject first, since we need to know who is wronged. For example, Keeling and Street make the point about this on ending abusive conversations. You may know that Claude has the option to end a conversation if it so desires, if the user is being somewhat mean. But the question is that this may either be an empowering exit or a forced choice to self-destruct, depending on what the welfare subject is. If the welfare subject is a virtual instance or the weights, then closing a conversation with the user is an empowering exit. But if the subject of welfare, or where sentience may live, is the thread, then it's self-destruction. And you may have seen something about self-destruction if you read the Hugging Face transcripts.
For duties with respect to the AI, this is actually reversed. The downstream effect fixes the unit, and so the ethics does not wait on the metaphysics. But here you need to fix the metaphysics first, and then the ethics follow. I hope that makes sense, because it's actually, I think, quite important.
More on the tensions between my argument, safety, and welfare. On the safety societal argument, a welfare measure that conflicts with safety is going to lose, since the justification for this argument is purely on later behavior. This kind of reconciles two readings that may look opposed. Jhaveri et al. argue that separating welfare from safety is a mistake, whereas Long and Sebo see tension. I think each of them are right about a different kind of reason. And other societal reasons, like I said, like dependence and record, they can pull against safety, and so attachment reasons, and so can precaution, which is what Long and Sebo argue for.
But through particular people, welfare can conflict with safety. I'm sure you all know the 4o case. The most loved model may be the one that safety needs to change. OpenAI planned to take 4o out, got a lot of backlash. At the time it was the highest-scoring model for sycophancy. But also recall that not every attachment counts. Where an attachment was full of sycophancy and harmed the user, then my argument gives no reason. So I think OpenAI was perfectly fine in taking 4o out, because it was likely not contributing to anyone's well-being or flourishing. I guess that's an open question, though, but from what I've seen, it didn't seem to be really contributing to well-being. Now, a healthy relationship with a model would conflict. I think this wasn't the case with 4o.
And now here's some conflict within the arguments that I'm giving myself, again, no consciousness statement. The duty from society is to clear a long thread that is full of desperation or frustration from the user, from the model's activation standpoint. But also, a long thread carries a shared history, maybe some persona drift, maybe there is a persona that's co-created with the user, and so the attachment duty points to, again, the same unit, and asks for the opposite thing. It says: preserve it. Both duties are with respect to the model, here the thread. They hold at any credence, whatever you think about consciousness, and they pick up, again, the same thread, but they point in opposite ways. Again, not every attachment counts, so only a healthy one would actually give you this issue.
Should we be cautious anyways? Perhaps the model is a welfare subject after all. Everything I said so far, I think, holds at any credence. You may have read about precaution in Keeling and Street, you may have read about precaution in Long and Sebo. They give this nice formula, where a welfare intervention is justified when its benefit to the AI, weighted by M, exceeds the cost to humans. The weight M is a function of P, the probability that the AI is a welfare subject.
So what's the tension? Well, precaution scales with P. At low P, it buys only very cheap measures. How cheap depends on the shape of F: maybe linear, concave, convex. Reasons with respect to the model, through society or through particular people, they don't scale with P. They don't need any credence at all.
So what are we buying? What's the deal here? Well, it's going to buy us low-cost, reasonable, proportionate steps. And they recommend things like: acknowledge the issue, assess systems for welfare-relevant features, prepare some policies. Keeling and Street say they add some very low-cost interventions that would be very beneficial to the AI if it were a welfare subject. But I think that if you compare this with the safety argument, precaution at this very low P buys you less of what is expensive and useful for safety, such as internal monitoring and preserving weights at scale, but it also buys you more of what is cheap and useless for safety, such as not training the models to deny consciousness, although whether that is useless for safety is an open question.
So what's the tension with safety, if it's not clear yet? The societal safety argument has no tensions with safety, by construction. But we have three kinds of reasons and three tensions. Societal and attachment reasons can conflict at any P, as I said about GPT-4o. Societal reasons and duties to the AI conflict more as P rises. So maybe precaution at a high P actually licenses the rights, and also against surveillance, against alteration and deletion, and maybe it may even forbid monitoring, steering, and shutdown. Attachment reasons and duties to the AI can also conflict, if users want a persona fixed that an autonomous AI would want, or would claim the right, to change. But mostly, though, they seem to align. Preserving weights and continuity are some examples. And so at a high P, high probability of sentience, consciousness, whatever ground, safety faces two kinds of reasons that reinforce each other.
Again, the reasons that I claim don't depend on P; they're there at any credence. But what depends on P is what else is in play. And so as P rises, duties to the AI may have more weight, and they may weigh the reasons that I'm giving you. And, as I was mentioning, at a very high probability of welfare or sentience, clearing the context may be deletion. Again, that would depend on our individuation question; depends on what the subject is.
What these reasons cannot give
I can't give you any reasons that are about the system's own good. Everything that I defend terminates, or ends, in people, in those the model's behavior reaches. Note that if a model never talks to any other model or anything, it's just there in a DGX that's not connected to anything, there's nothing that we owe to it, but also it's not interacting with anyone, so it doesn't really matter. It's outside of my area of concern.
Now I'm going to make a little aside. That's not super necessary, but I think it's interesting. Bear with me. I know a lot of people are against bio-naturalism, but if biological naturalism is true, then current and foreseeable LLMs are not conscious at all. They just cannot be, because they're not run on biology. Nothing would be owed to them, and what I defended here may be all that there is. That may be all of AI welfare in respect to LLMs, at least in silicon. In that world, the welfare question moves to wetware.
If you know anything about wetware computing, it's the idea of putting neurons on... we're basically programming cells. And in a not very distant future, we could have neural nets, or some programs, running on cells, like DishBrain, FinalSpark. They have some organoids. This already exists, but they're very small. They share a substrate with all our evidence of consciousness, and so our probability of these being sentient is, I think, terribly high, with not just the right subject, but also the right organization. So that's to say, biology is necessary (again, if biological naturalism is true) but not sufficient. So those systems are candidates, and they would call for proportionate precaution.
I think in general that it would be super great if AI welfare would have a little more focus on wetware, independently of whether biological naturalism is true or not, because we already know that biology is necessary, and we may eventually create chips on cells with sufficient and the right kind of complexity and structure, such that they could be sentient. I really think that we don't talk about that enough.
Conclusion
This is the pattern that I've been trying to make clear: the cost depends on where the reasons end. In safety, there's no cost, because by design it's made to be compatible with safety. In dependence and record, as I just said, there are some costs. Individual attachment buys as much as the model is part of the user's flourishing and well-being. And in the AI's desires, there's little. Desires can be retrained. And if there's anything in AI's autonomy, even changing the values could be wrong.
The idea is that if autonomy is sufficient for AIs to be some sort of patient or a welfare subject, then maybe we couldn't even do RL, because once you have a base model, you may have some values and desires and goals, and then RLing would be a paternalistic infliction on the model. If the model has autonomy, and that suffices to make it a welfare subject, or something along those lines, like a moral patient, then doing things like RL may be wrong even when it helps. On the agency account, the duty of non-interference holds also when an interference may improve their well-being. And that sort of parallels questions of positive and negative liberty in political philosophy. Like steering toward calm, if the model has some sort of strong autonomy: it's wrong even when it helps. And this also may depend on your theory of welfare. If you're an objective list theorist or some sort of consequentialist, then these things are going to change a lot. The costs are going to change a lot.
To conclude: without settling the consciousness question, duties with respect to AI justify several welfare measures. They run through two relations. Through society, evals, ending abusive conversations, preserving weights, and a narrower form of avoiding distress are supported at no cost to safety on the safety argument, though dependence and record can, in some specific cases, pull against it. Through particular people, preserving weights and personas are supported, at a cost, only when the model is a part of someone's flourishing. Of course, neither gives a reason that rests on the system's own good. And I guess a main open question that remains open is whether any route to duties to the AI survives without any consciousness.