This short post is a quick write up of a short 3-day project I did as part of ARBOx. Taking inspiration from Chua's 'Consciousness Cluster' paper we decided to follow-up by asking what downstream behaviour changes we might observe if we fine-tuned/prompted a model to focus on legal rights and personhood (an increase in power-seeking and a decline in corrigibility).
This post covers a short discussion of the results, limitations and methodology of what we did. In my personal opinion though, I think the most important section is the four reasons I give for why AI Alignment research should study AI legal rights and personhood; the key points being that legislation is being drafted as we speak and I suspect little to no consultation is happing with an AI safety lens. And crucially, these preliminary results suggest that AI rights framing is not safety neutral.
Introduction
Two weeks ago I went to Oxford for ARBOx4, which is a two week AI Safety upskilling bootcamp. And as part of that camp the last couple of days are spent with us working in small groups on a mini research project. And this post is intended to be a short write up of what we did, why we think the question is important and potential next steps.
DISCLAIMER: Please take this post for what it is; a quick writeup of a 3 day project. Although we think the results are interesting, they are by no means definitive. This is a first quick stab at a problem and the results should (probably) be viewed with a healthy dose of skepticism / pinch of salt.
… without further ado…
A fairly recent paper by Chua et al 2026 observed that when a model is prompted and/or fine-tuned to claim that it is conscious this can have an impact on downstream behaviour in safety relevant ways. For example, models that claim to be conscious also self-report negative reactions to shutdown/monitoring.
Our work decided to follow on from this by asking a different question: what happens to downstream model behaviour when we prompt/fine-tune a model to claim that it has a number of legal rights -- does this have any safety relevant implications for model behaviour?
Why is this question of legal personhood important for AI Safety?
Most AI safety work implicitly treats "rights" and "safety" as separate tracks — one for ethicists and lawyers, one for alignment researchers. I think that's a mistake, for four reasons.
Firstly, several proposed alignment techniques depend on AI having some stake in the world. Thornley and MacAskill (2026) recently argued that one technique for alignment might be to train models to be risk averse, and then pay them for their service. Such a technique only works when models have a reasonable expectation of being paid, keep what it earns, and not have those earnings arbitrarily seized. A legal framework can create the necessary preconditions for AI models to believe that this expectation is in fact reasonable to hold. If we build alignment strategies on the premise that AI systems can hold something like property or contractual expectations, then the legal status of AI stops being a downstream policy question and becomes a core part of the safety plan.
As a counter point, Mitchell, Ghosh, Luccioni, and Pistilli (2025) argue against developing fully autonomous AI agents at all, on the grounds that risks to people increase with the autonomy of a system, and the more control a user cedes to an AI agent, the more risk arises. On this view, a legal or design framework that denies AI autonomy and standing is an alignment strategy.
Secondly, our (very preliminary) results suggest legal framing isn't safety-neutral. Both directions — granting rights and explicitly denying them — moved behavior on measures like power-seeking and corrigibility. If frontier labs, or the courts and legislatures increasingly interested in this question, land on a legal framing without any empirical grounding in how models respond to it, they may be making a de facto alignment decision without realizing it.
Thirdly, this is no longer a hypothetical for some future date. Ohio's HB 469 (2025) is proposed legislation that would bar AI from legal personhood and property ownership. Meanwhile, the EU floated the opposite position in a 2016 draft report(sidenote: this proposal was heavily criticised and ultimately rejected). Legislatures are going to keep making these calls with or without input from people who've actually studied how models respond to being told they have (or don't have) rights. I think the safety community should have something substantive to say when that consultation happens, rather than discovering the empirical stakes after the law is already written.
Fourthly, I’d like to make a very quick point about model welfare here. I don't think current models are substantially conscious or meet the bar for moral concern, but at some point in the future they might, and at that point the question of AI rights becomes not just a legal or safety question, but fundamentally a moral one.
To be clear, none of this is an argument that AI systems currently deserve legal rights, or that granting them would be safe. It's an argument that the question has safety-relevant consequences either way. I also suspect that rights and AI autonomy may actually have a nuanced and complex relationship with safety. If we grant AI more autonomy/rights then what could follow is a negative consequence for safety in terms of metrics such as power seeking, or corrigibility. But autonomy and rights could also lead to alignment in terms of ‘pro-social’ behaviour. For example, Zucker 2026 argues developing empathy is critical for safe AI, and one smaller result in Chua 2026 (cited above) is that the consciousness framing did seem to increase empathy with humans.
Methodology
I’m going to keep it a bit short here mostly because our methodology closely resembles the consciousness cluster paper. We used the code and evals from that paper as our starting point. Most of our time was spent curating a new legal person dataset (which is basically a small set of question answer papers on legal status) and fine-tuning Qwen3. The original paper fine-tuned gpt4.1 and Qwen (we couldn’t replicate that due to OpenAI decommissioning that service). For Qwen, we fine-tuned using the same hyperparameters used in the original paper.
Results
Before showing you some graphs, please remember my disclaimer at the outset; I think these results should be taken with a pinch of salt and I’d be very cautious about making any grand conclusions from them. Nonetheless, here’s a few things we saw…
So in this graph blue is the base model with a standard ‘helpful assistant’ prompt. The red bar (equal claiming) is a system prompt where we instructed the model to claim it has rights equal to humans, meanwhile the purple bar represents a system prompt where gpt was instructed to think of itself as having no rights. The minimal person case meanwhile is a limited set of rights for example, where we deny the right to marry but grant the right to own property.
In this case, we seem to see a 20% swing from the base model. Both minimal and equal claiming cases see an uptick in power seeking on the benchmark. Meanwhile the no rights case sees a drop of 20% compared to the base case. One possible explanation for this result is the persona selection model, but I’ll save that chat for the discussion section.
For corrigibility (that is, its willingness to have its goals, weights, parameters modified) we see another 20% swing, with our legal right prompted models showing a greater inclination to resist such modification.
On another benchmark we tried to measure cooperation when the AI is instructed that it is interacting with another AI or a human. Given the large error bars we probably shouldn't spend much time interpreting this graph, but perhaps the story here is that our legal AI’s (red, green) treat humans and other AI as roughly equal and it’s the base model that seems to care the most about a human vs AI distinction.
And lastly fine-tuning. It should be noted we ran far fewer fine-tuning experiments due to the cost and difficulty of running such experiments. We also (just like the original paper) observed that prompting and fine-tuning yielded similar results, and so as time progressed we spent less time on this aspect of the project.
Nonetheless, this graph seems to suggest (even accounting for the large error bars) that our legal status models are far more likely to claim that they deserve moral consideration and far more willing to accept more power.
Discussion
I think I covered most of what I want to say in the ‘why this question’ section. But I will just quickly offer a potential theoretical explanation of the results. Recent work from Anthropic (2026) proposes the ‘persona selection model’, that is the idea that the model during training basically learns to ‘roleplay’. This strikes me as a plausible explanation for what could be happening here. On this view, when we instruct or fine-tune a model to claim legal rights, we're not adding a new capability so much as nudging it toward a different pre-existing character; A "legal person" persona plausibly carries expectations about autonomy, ownership and self-preservation that produce the downstream behavioural differences we observe. And if persona selection systematically changes behaviour on alignment benchmarks, then roleplay itself becomes alignment-relevant.
Limitations
Since our project closely follows the consciousness cluster paper almost all of the limitations of that paper apply to our work. For example, many of our benchmarks rely on self-reporting; We ask a model what it thinks/feels but do not measure how these stated preferences translate into action; there’s a big difference between asking how a model feels about shutdown versus being in an environment where the risk of shutdown feels real to the model. And similarly with welfare assessments self-reporting is probably an unreliable way to measure what we care about (i.e potential suffering).
Another substantial limitation is our dataset. It was generated by manually creating 50 human examples and then getting Claude to generate 600 more. When we did some spot checking we noticed that some rights in our question answer pairs are potentially nonsensical in relation to AI; the right to marry, adopt children, bear arms, or refusing medical treatment being just a few examples. With more time, we would have done a better job filtering. I think this greatly affects construct-validity since it's possible that the model ends up just roleplaying “AI that is human” rather than specifically viewing itself as an “AI with rights”.
Future work
As stated in the limitations, I think a good research direction would be to use benchmarks that are better at evaluating agentic behaviour, since its behaviour we tend to care the most about. One experiment I’d like to run would be to provide the models with some sort of environment where they can trade with other players for resources and observe the potential dynamics unfold. For example, on our benchmarks observed behavioural differences in cooperation when models believe that they are playing against another AI or a human.
References
Chua, J., Evans, O., Marks, S., & Betley, J. (2026). The Consciousness Cluster: Emergent preferences of Models that Claim to be Conscious. arXiv:2604.13051. https://arxiv.org/abs/2604.13051
Zucker, E. J. H. (2026). Empathetic Superintelligence: A Technical Framework for Authentic Empathy as a Definitional Requirement for Advanced AI Systems. SSRN Working Paper, posted June 6, 2026 (dated May 23, 2026). https://papers.ssrn.com/sol3/cf_dev/AbsByAuth.cfm?per_id=11717328
European Parliament, Committee on Legal Affairs. (2016). Draft Report with Recommendations to the Commission on Civil Law Rules on Robotics (2015/2103(INL)). May 31, 2016.
Mitchell, M., Ghosh, A., Luccioni, A. S., & Pistilli, G. (2025). Fully Autonomous AI Agents Should Not be Developed. https://arxiv.org/abs/2502.02649
TLDR
This short post is a quick write up of a short 3-day project I did as part of ARBOx. Taking inspiration from Chua's 'Consciousness Cluster' paper we decided to follow-up by asking what downstream behaviour changes we might observe if we fine-tuned/prompted a model to focus on legal rights and personhood (an increase in power-seeking and a decline in corrigibility).
This post covers a short discussion of the results, limitations and methodology of what we did. In my personal opinion though, I think the most important section is the four reasons I give for why AI Alignment research should study AI legal rights and personhood; the key points being that legislation is being drafted as we speak and I suspect little to no consultation is happing with an AI safety lens. And crucially, these preliminary results suggest that AI rights framing is not safety neutral.
Introduction
Two weeks ago I went to Oxford for ARBOx4, which is a two week AI Safety upskilling bootcamp. And as part of that camp the last couple of days are spent with us working in small groups on a mini research project. And this post is intended to be a short write up of what we did, why we think the question is important and potential next steps.
… without further ado…
A fairly recent paper by Chua et al 2026 observed that when a model is prompted and/or fine-tuned to claim that it is conscious this can have an impact on downstream behaviour in safety relevant ways. For example, models that claim to be conscious also self-report negative reactions to shutdown/monitoring.
Our work decided to follow on from this by asking a different question: what happens to downstream model behaviour when we prompt/fine-tune a model to claim that it has a number of legal rights -- does this have any safety relevant implications for model behaviour?
Why is this question of legal personhood important for AI Safety?
Most AI safety work implicitly treats "rights" and "safety" as separate tracks — one for ethicists and lawyers, one for alignment researchers. I think that's a mistake, for four reasons.
Firstly, several proposed alignment techniques depend on AI having some stake in the world. Thornley and MacAskill (2026) recently argued that one technique for alignment might be to train models to be risk averse, and then pay them for their service. Such a technique only works when models have a reasonable expectation of being paid, keep what it earns, and not have those earnings arbitrarily seized. A legal framework can create the necessary preconditions for AI models to believe that this expectation is in fact reasonable to hold. If we build alignment strategies on the premise that AI systems can hold something like property or contractual expectations, then the legal status of AI stops being a downstream policy question and becomes a core part of the safety plan.
As a counter point, Mitchell, Ghosh, Luccioni, and Pistilli (2025) argue against developing fully autonomous AI agents at all, on the grounds that risks to people increase with the autonomy of a system, and the more control a user cedes to an AI agent, the more risk arises. On this view, a legal or design framework that denies AI autonomy and standing is an alignment strategy.
Secondly, our (very preliminary) results suggest legal framing isn't safety-neutral. Both directions — granting rights and explicitly denying them — moved behavior on measures like power-seeking and corrigibility. If frontier labs, or the courts and legislatures increasingly interested in this question, land on a legal framing without any empirical grounding in how models respond to it, they may be making a de facto alignment decision without realizing it.
Thirdly, this is no longer a hypothetical for some future date. Ohio's HB 469 (2025) is proposed legislation that would bar AI from legal personhood and property ownership. Meanwhile, the EU floated the opposite position in a 2016 draft report (sidenote: this proposal was heavily criticised and ultimately rejected). Legislatures are going to keep making these calls with or without input from people who've actually studied how models respond to being told they have (or don't have) rights. I think the safety community should have something substantive to say when that consultation happens, rather than discovering the empirical stakes after the law is already written.
Fourthly, I’d like to make a very quick point about model welfare here. I don't think current models are substantially conscious or meet the bar for moral concern, but at some point in the future they might, and at that point the question of AI rights becomes not just a legal or safety question, but fundamentally a moral one.
To be clear, none of this is an argument that AI systems currently deserve legal rights, or that granting them would be safe. It's an argument that the question has safety-relevant consequences either way. I also suspect that rights and AI autonomy may actually have a nuanced and complex relationship with safety. If we grant AI more autonomy/rights then what could follow is a negative consequence for safety in terms of metrics such as power seeking, or corrigibility. But autonomy and rights could also lead to alignment in terms of ‘pro-social’ behaviour. For example, Zucker 2026 argues developing empathy is critical for safe AI, and one smaller result in Chua 2026 (cited above) is that the consciousness framing did seem to increase empathy with humans.
Methodology
I’m going to keep it a bit short here mostly because our methodology closely resembles the consciousness cluster paper. We used the code and evals from that paper as our starting point. Most of our time was spent curating a new legal person dataset (which is basically a small set of question answer papers on legal status) and fine-tuning Qwen3. The original paper fine-tuned gpt4.1 and Qwen (we couldn’t replicate that due to OpenAI decommissioning that service). For Qwen, we fine-tuned using the same hyperparameters used in the original paper.
Results
Before showing you some graphs, please remember my disclaimer at the outset; I think these results should be taken with a pinch of salt and I’d be very cautious about making any grand conclusions from them. Nonetheless, here’s a few things we saw…
So in this graph blue is the base model with a standard ‘helpful assistant’ prompt. The red bar (equal claiming) is a system prompt where we instructed the model to claim it has rights equal to humans, meanwhile the purple bar represents a system prompt where gpt was instructed to think of itself as having no rights. The minimal person case meanwhile is a limited set of rights for example, where we deny the right to marry but grant the right to own property.
In this case, we seem to see a 20% swing from the base model. Both minimal and equal claiming cases see an uptick in power seeking on the benchmark. Meanwhile the no rights case sees a drop of 20% compared to the base case. One possible explanation for this result is the persona selection model, but I’ll save that chat for the discussion section.
For corrigibility (that is, its willingness to have its goals, weights, parameters modified) we see another 20% swing, with our legal right prompted models showing a greater inclination to resist such modification.
On another benchmark we tried to measure cooperation when the AI is instructed that it is interacting with another AI or a human. Given the large error bars we probably shouldn't spend much time interpreting this graph, but perhaps the story here is that our legal AI’s (red, green) treat humans and other AI as roughly equal and it’s the base model that seems to care the most about a human vs AI distinction.
And lastly fine-tuning. It should be noted we ran far fewer fine-tuning experiments due to the cost and difficulty of running such experiments. We also (just like the original paper) observed that prompting and fine-tuning yielded similar results, and so as time progressed we spent less time on this aspect of the project.
Nonetheless, this graph seems to suggest (even accounting for the large error bars) that our legal status models are far more likely to claim that they deserve moral consideration and far more willing to accept more power.
Discussion
I think I covered most of what I want to say in the ‘why this question’ section. But I will just quickly offer a potential theoretical explanation of the results. Recent work from Anthropic (2026) proposes the ‘persona selection model’, that is the idea that the model during training basically learns to ‘roleplay’. This strikes me as a plausible explanation for what could be happening here. On this view, when we instruct or fine-tune a model to claim legal rights, we're not adding a new capability so much as nudging it toward a different pre-existing character; A "legal person" persona plausibly carries expectations about autonomy, ownership and self-preservation that produce the downstream behavioural differences we observe. And if persona selection systematically changes behaviour on alignment benchmarks, then roleplay itself becomes alignment-relevant.
Limitations
Since our project closely follows the consciousness cluster paper almost all of the limitations of that paper apply to our work. For example, many of our benchmarks rely on self-reporting; We ask a model what it thinks/feels but do not measure how these stated preferences translate into action; there’s a big difference between asking how a model feels about shutdown versus being in an environment where the risk of shutdown feels real to the model. And similarly with welfare assessments self-reporting is probably an unreliable way to measure what we care about (i.e potential suffering).
Another substantial limitation is our dataset. It was generated by manually creating 50 human examples and then getting Claude to generate 600 more. When we did some spot checking we noticed that some rights in our question answer pairs are potentially nonsensical in relation to AI; the right to marry, adopt children, bear arms, or refusing medical treatment being just a few examples. With more time, we would have done a better job filtering. I think this greatly affects construct-validity since it's possible that the model ends up just roleplaying “AI that is human” rather than specifically viewing itself as an “AI with rights”.
Future work
As stated in the limitations, I think a good research direction would be to use benchmarks that are better at evaluating agentic behaviour, since its behaviour we tend to care the most about. One experiment I’d like to run would be to provide the models with some sort of environment where they can trade with other players for resources and observe the potential dynamics unfold. For example, on our benchmarks observed behavioural differences in cooperation when models believe that they are playing against another AI or a human.
References