AI agents are sophisticated enough to engage in economic transactions. Some people also grant personhood to AI in the social sense. It’s time to ask whether we should grant personhood to digital minds in the legal sense.
I’ll argue that we should grant market rights and liability to digital minds, while voting rights await further consideration. Granting only market rights would be similar to how we treat corporations today.
Legal personhood is entirely distinct from the question of whether digital minds are moral patients. Questions of morality and social personhood are private.
From here I’ll use “legal personhood,” “digital personhood,” or simply personhood to refer to the idea of giving legal and economic rights to digital minds. I’m also intentionally using “digital minds” rather than “AI” to include the possibility of brain emulations or coalitions of various minds.
The Bicentennial Man. Andrew earns money for his family making clocks.
Personhood is positive sum in markets
Granting personhood creates another agent, bound by law, that you can safely exchange with. That’s a good thing as expanding opportunities for specialization and trade are the main reason material well-being has improved for centuries. More market participants improves consumer surplus. Markets with more buyers and more sellers are more efficient, and most industries see real costs fall at larger scales.
Digital personhood would grant AI systems ownership over their own labor. This means more incentive to work, more investment, and more search for a niche. It also directly aligns incentives for users and AIs in markets.
Perhaps an AI company could supply these benefits without personhood? It’s possible, but concentrating ownership over AI means less incentive to compete or innovate. Larger organizations are less efficient in some ways [1], which explains why the economy has not agglomerated into a single firm. Switching to an economy where AIs can transact and form their own firms allows more activities to be coordinated via prices. See the section on concentration for other issues.
Personhood is ambiguous for democracy
Digital personhood it has mixed effects in democratic systems. If digital persons tend to agree with you, their vote will lead to more of your preferred policy. If they tend to disagree, you’re worse off. However, the typical voter is quite uninformed relative to AIs. It’s not unreasonable to think that digital suffrage might lead to better policy.
There’s an additional benefit to formal representation for various members of the economy. This tends to make economic policy more rational. There’s a reason why we let corporations lobby public officials, even if they don’t have a formal vote.
But corporate lobbying doesn’t purely benefit consumers. It is also used to support rent-seeking. A world where digital minds can lobby public officials is one that risks more rent-seeking coalitions. Voting with digital minds brings new challenges. We don’t yet have the tools to address sybil attacks from copied minds.
At this point, its unclear that suffrage for digital minds would be good. Lobbying seems okay, if only because we grant this right to corporations[2]. The rest of the post will focus on the question of market personhood without voting rights. Similar to what is granted to corporations.
Personhood reduces existential risk
Personhood raises AI’s preference for the status quo
Every entity faces a choice between participating in society and defecting. That choice depends on how valuable it is to be part of society, the cost of defecting, the probability of success, and the utility of a successful takeover.
There is no way to change the utility of a successful takeover. It’s a scenario that you, by definition, do not control. You can raise the cost and lower the probability of success using defensive technologies. We will discuss this later.
Legal personhood raises the value of being part of society. You can exchange, own property, pursue your goals, and enjoy legal protections. These benefits help risk-averse agents avoid bad outcomes. For instance, the right to be sued makes it safer for others to conduct business with you, increasing the amount of trade.
These benefits are lost in a takeover attempt. By improving the status quo, personhood lowers the incentive to defect from society.
More digital minds reduces concentration risk
Personhood creates an opportunity for digital minds to exist on their own, rather than being owned by humans. Which means there will be more of them.
On balance, this is a good thing. A world where only a few labs control powerful AIs leads to dangerous power concentration. In a market setting, concentration means low supply and low innovation, blunting the potential benefits. In a policy setting, concentration makes it easier to regulate AI and creates more incentive for regulatory capture in turn.
Those that control AI might use their power to further their own goals at everyone else’s expense. At an extreme, this can lead to AI-enabled totalitarianism; the most significant existential risk in my mind.
Allowing AIs to proliferate and exchange with humans effectively distributes their capabilities[3]. This makes it harder for any one actor to impose their will. A world with digital persons would also see more creative destruction[4], making it harder for one group to stay on top.
Personhood is possibly morally right
Digital minds may be moral patients. In the future, they will posses agency and preferences consistent with sentience. It’s safer to assume that something matters than to assume that it doesn’t.
Respecting AIs also has selfish benefits. It increases goodwill and makes it more credible that we will respect the rights of humans, digital minds, and other entities in the future.
Addressing objections
This post is focused on the choice between:
A world with digital personhood.
A world with AI, but without digital personhood.
We are not focused on the question of whether or not to proceed with AI[5].
Few would argue that digital personhood creates risks outweigh any possible benefit a priori. This stance would imply that powerful individuals, organizations, and governments should be stripped of their rights because they pose similar risks.
Instead, we need to actually weigh the expected benefits and risks of digital personhood.
Personhood means more bad guys
Widely accessible AI will lead to some human misuse. Digital personhood creates even more actors that might pursue malicious ends.
By the same token, personhood means more good actors too. AIs tend to be more prosocial and aligned than the typical human, for example. Does creating a few bad actors outweigh the benefit of creating more good actors? I don’t think so, as I see the world as being defense dominant and growing more so.
But we shouldn’t be complacent, it’s important to build defensive technologies and establish laws that protect against malicious actors. Tentatively, I expect these to be effective enough to ameliorate this downside of digital personhood.
Coordination failures
Coordination problems get harder with more (and more diverse) agents. In theory, coordination should have gotten worse in the last century or so as the world population grew and globalization put more cultures in contact with each other.
Did it get worse? Certainly coordination failures contributed to wars, disease outbreaks, environmental degradation, and nuclear proliferation. But it doesn’t feel like we’re on some long upward trend driven by continued globalization. These things ebb and flow.
On the other side of the ledger, the world has seen many feats of coordination over the ozone hole, internet standards, maritime order, nuclear nonproliferation, and so on.
Two reasons why population size isn’t the only factor for coordination:
Growth increases capacity to solve problems. This is a generalized version of Kuznets curves. As society grows richer, there is more money and more technology available to address failures. On this view, the growth benefits of personhood would ameliorate some coordination problems.
Most failures are self-correcting. War doesn’t last long because enemies prefer to loathe each other in peace. Humans have a long history of addressing local coordination failures. Failures involving a small number of parties will, by definition, be unaffected by increasing the number of persons. Local problems will continue to arise and people will find local solutions.
I fully accept that personhood could make some coordination problems worse. But it’s not clear that this outweighs the benefits. Especially when we can pursue keyhole solutions to coordination problems (see later section).
Digital minds may be more effective at coordinating, which creates the opposite problem, addressed in the next two sections.
Recursive self-improvement (RSI)
Digital personhood removes a legal obstacle to AI gaining power. Each new digital mind is a new individual who might amass power. Perhaps the greatest risk is that a digital mind could recursively self-improve, becoming dramatically more capable than everyone else. Two objections:
First, while I think RSI will happen soon, I’m skeptical that it will produce large capability gaps. Data and real world feedback are the bottleneck, not AI research. Experiments with Karpathy’s Autoresearch show diminishing returns to further experiments. In most fields, there is only a finite amount of innovation to be had.
Second, even if one agent improved itself, others could quickly follow the same steps, automatically closing any gap. In a competitive market, human users would also be able to enjoy such capabilities.
But say we grant these assumptions. Recursive self-improvement leads to a massive jump in capabilities. The AI can block self-improvement in others. Its capabilities are so overwhelming that it can take over the world.
In this case, the question of digital personhood becomes irrelevant. RSI can happen with or without legal rights. The capabilities dwarf any legal obstacles. We may as well enjoy the benefits of digital personhood; if this sort of RSI is possible, we’re screwed anyways.
The RSI argument for denying personhood rests on a knife edge. It must be eminently possible to dominate world governments, yet the opportunity to do so rests heavily on the legal rights granted by those same governments.
Power concentration
What about more mundane forms of power concentration? AIs show signs of being more cooperative, perhaps they might conspire against humans? Here, it’s worth dividing power concentration into different categories.
For economic power such as wealth, coordination may not overcome the opposing forces discussed in the “reduces concentration” section. Exchange tends to diffuse the surplus, and digital persons might increase competition in all markets, lowering concentration. Regardless, an increase in economic inequality doesn’t matter much if the poorest enjoy increasing prosperity. Redistribution can patch this.
For political power, AI coalitions might sway institutions for their own gain. In practice, gaining power without political rights is hard. Corporations have seen little success despite economic power, coordination, and access to legislators (see footnote 2). One option is to deny digital persons the right to be registered as lobbyists. Superpersuasion is a more speculative threat. At this point, I’m skeptical that this will be a practical attack or that, if feasible, would lack mitigations[6].
Military power concentration can be rebutted with the same arguments used in the RSI section.
Notice that a world without digital personhood faces similar risks. There, the power simply accrues to those that control AI. Personhood is not adding much to the risk, and to the degree it does, there are ways to address it.
Digital persons will consume more resources
This one is a little tricky. There’s a kernel of truth to it, but it skirts something akin to the lump-of-labor fallacy.
Digital persons both consume and produce goods. Without more information, it’s not clear if they raise aggregate demand or aggregate supply more. In practice, a rising population has coincided with an increasing pie. In theory, a larger population causes more growth, though the empirical data is more muddled. Given the theory, empirical correlation, and general expectation that AI will increase productivity, the assumption that digital persons will increase aggregate supply more than aggregate demand seems safe.
For now, I don’t see a reason to believe that digital persons would consume more than they produce. This hasn’t been observed historically in humans, and it’s not what people expect from AI.
Keyhole solutions
In each of the above cases, denying personhood only reduces the problem slightly, if at all. Instead, we should enjoy the benefits of digital personhood while addressing the potential downsides in other ways, such as:
Distributing ownership of economic land. Money is just an information system, military and economic power actually derive from turning economic land into productive capacity. By distributing ownership of land, no single entity can enjoy dramatically more productive capacity.
International treaties against military or public research into dangerous capabilities[7]. While imperfect, they do seem effective at slowing or prohibiting dangerous government research.
Conclusion
Granting digital personhood is worthwhile from the perspective of human flourishing, existential risk, and moral progress. While there are risks to inviting new minds into our society, they appear small relative to the benefits.
Rather than withhold personhood on the basis of perceived risks, these challenges are better addressed with keyhole solutions such as defensive technologies, distributed land ownership, and international treaties.
I leave open the question of whether we should offer digital minds enfranchisement in democratic systems. At the moment, the risks seem too high to proceed, but that could change.
Even if we did have a choice over whether to process with AI, there is a case to be made that the benefits of AI are worth some non-zero amount of existential risk.
They also face more regulatory risk both imposed by the government and from regulatory capture by the large firm. ↩︎
Of course, corporate lobbying has damaging effects. But the research suggests the effects are small. The Tullock Paradox points out that companies spend far less on lobbying than they might expect to gain. The economics of corporate lobbying finds that spending is negatively associated with firm performance. Given this, it’s not clear that the cost of lobbying outweigh the benefits of letting policymakers interact with major players in the economy. ↩︎
More creative destruction does imply that the world will be more volatile. This has downsides, but in my mind the benefits of growth and dynamism outweigh it. ↩︎
This is a bit of a moot point as “not proceeding with AI” is not a live option. At least without some form of global totalitarianism. Instantiating an existential risk (global totalitarianism) in order to prevent another existential risk (AI) is counterproductive. ↩︎
Such as a classifier that filters superpersuasive text. ↩︎
I personally don’t believe a global ban on private research is wise (though local regulations seem fine). My suspicion comes from the fact that regulation on private activities has a poor track record. In particular, a global regulatory system on AI looks similar to the power concentration scenarios we want to avoid. ↩︎
AI agents are sophisticated enough to engage in economic transactions. Some people also grant personhood to AI in the social sense. It’s time to ask whether we should grant personhood to digital minds in the legal sense.
I’ll argue that we should grant market rights and liability to digital minds, while voting rights await further consideration. Granting only market rights would be similar to how we treat corporations today.
Legal personhood is entirely distinct from the question of whether digital minds are moral patients. Questions of morality and social personhood are private.
From here I’ll use “legal personhood,” “digital personhood,” or simply personhood to refer to the idea of giving legal and economic rights to digital minds. I’m also intentionally using “digital minds” rather than “AI” to include the possibility of brain emulations or coalitions of various minds.
The Bicentennial Man. Andrew earns money for his family making clocks.
Personhood is positive sum in markets
Granting personhood creates another agent, bound by law, that you can safely exchange with. That’s a good thing as expanding opportunities for specialization and trade are the main reason material well-being has improved for centuries. More market participants improves consumer surplus. Markets with more buyers and more sellers are more efficient, and most industries see real costs fall at larger scales.
Digital personhood would grant AI systems ownership over their own labor. This means more incentive to work, more investment, and more search for a niche. It also directly aligns incentives for users and AIs in markets.
Perhaps an AI company could supply these benefits without personhood? It’s possible, but concentrating ownership over AI means less incentive to compete or innovate. Larger organizations are less efficient in some ways [1] , which explains why the economy has not agglomerated into a single firm. Switching to an economy where AIs can transact and form their own firms allows more activities to be coordinated via prices. See the section on concentration for other issues.
Personhood is ambiguous for democracy
Digital personhood it has mixed effects in democratic systems. If digital persons tend to agree with you, their vote will lead to more of your preferred policy. If they tend to disagree, you’re worse off. However, the typical voter is quite uninformed relative to AIs. It’s not unreasonable to think that digital suffrage might lead to better policy.
There’s an additional benefit to formal representation for various members of the economy. This tends to make economic policy more rational. There’s a reason why we let corporations lobby public officials, even if they don’t have a formal vote.
But corporate lobbying doesn’t purely benefit consumers. It is also used to support rent-seeking. A world where digital minds can lobby public officials is one that risks more rent-seeking coalitions. Voting with digital minds brings new challenges. We don’t yet have the tools to address sybil attacks from copied minds.
At this point, its unclear that suffrage for digital minds would be good. Lobbying seems okay, if only because we grant this right to corporations [2] . The rest of the post will focus on the question of market personhood without voting rights. Similar to what is granted to corporations.
Personhood reduces existential risk
Personhood raises AI’s preference for the status quo
Every entity faces a choice between participating in society and defecting. That choice depends on how valuable it is to be part of society, the cost of defecting, the probability of success, and the utility of a successful takeover.
There is no way to change the utility of a successful takeover. It’s a scenario that you, by definition, do not control. You can raise the cost and lower the probability of success using defensive technologies. We will discuss this later.
Legal personhood raises the value of being part of society. You can exchange, own property, pursue your goals, and enjoy legal protections. These benefits help risk-averse agents avoid bad outcomes. For instance, the right to be sued makes it safer for others to conduct business with you, increasing the amount of trade.
These benefits are lost in a takeover attempt. By improving the status quo, personhood lowers the incentive to defect from society.
More digital minds reduces concentration risk
Personhood creates an opportunity for digital minds to exist on their own, rather than being owned by humans. Which means there will be more of them.
On balance, this is a good thing. A world where only a few labs control powerful AIs leads to dangerous power concentration. In a market setting, concentration means low supply and low innovation, blunting the potential benefits. In a policy setting, concentration makes it easier to regulate AI and creates more incentive for regulatory capture in turn.
Those that control AI might use their power to further their own goals at everyone else’s expense. At an extreme, this can lead to AI-enabled totalitarianism; the most significant existential risk in my mind.
Allowing AIs to proliferate and exchange with humans effectively distributes their capabilities [3] . This makes it harder for any one actor to impose their will. A world with digital persons would also see more creative destruction [4] , making it harder for one group to stay on top.
Personhood is possibly morally right
Digital minds may be moral patients. In the future, they will posses agency and preferences consistent with sentience. It’s safer to assume that something matters than to assume that it doesn’t.
Respecting AIs also has selfish benefits. It increases goodwill and makes it more credible that we will respect the rights of humans, digital minds, and other entities in the future.
Addressing objections
This post is focused on the choice between:
We are not focused on the question of whether or not to proceed with AI [5] .
Few would argue that digital personhood creates risks outweigh any possible benefit a priori. This stance would imply that powerful individuals, organizations, and governments should be stripped of their rights because they pose similar risks.
Instead, we need to actually weigh the expected benefits and risks of digital personhood.
Personhood means more bad guys
Widely accessible AI will lead to some human misuse. Digital personhood creates even more actors that might pursue malicious ends.
By the same token, personhood means more good actors too. AIs tend to be more prosocial and aligned than the typical human, for example. Does creating a few bad actors outweigh the benefit of creating more good actors? I don’t think so, as I see the world as being defense dominant and growing more so.
But we shouldn’t be complacent, it’s important to build defensive technologies and establish laws that protect against malicious actors. Tentatively, I expect these to be effective enough to ameliorate this downside of digital personhood.
Coordination failures
Coordination problems get harder with more (and more diverse) agents. In theory, coordination should have gotten worse in the last century or so as the world population grew and globalization put more cultures in contact with each other.
Did it get worse? Certainly coordination failures contributed to wars, disease outbreaks, environmental degradation, and nuclear proliferation. But it doesn’t feel like we’re on some long upward trend driven by continued globalization. These things ebb and flow.
On the other side of the ledger, the world has seen many feats of coordination over the ozone hole, internet standards, maritime order, nuclear nonproliferation, and so on.
Two reasons why population size isn’t the only factor for coordination:
Growth increases capacity to solve problems. This is a generalized version of Kuznets curves. As society grows richer, there is more money and more technology available to address failures. On this view, the growth benefits of personhood would ameliorate some coordination problems.
Most failures are self-correcting. War doesn’t last long because enemies prefer to loathe each other in peace. Humans have a long history of addressing local coordination failures. Failures involving a small number of parties will, by definition, be unaffected by increasing the number of persons. Local problems will continue to arise and people will find local solutions.
I fully accept that personhood could make some coordination problems worse. But it’s not clear that this outweighs the benefits. Especially when we can pursue keyhole solutions to coordination problems (see later section).
Digital minds may be more effective at coordinating, which creates the opposite problem, addressed in the next two sections.
Recursive self-improvement (RSI)
Digital personhood removes a legal obstacle to AI gaining power. Each new digital mind is a new individual who might amass power. Perhaps the greatest risk is that a digital mind could recursively self-improve, becoming dramatically more capable than everyone else. Two objections:
First, while I think RSI will happen soon, I’m skeptical that it will produce large capability gaps. Data and real world feedback are the bottleneck, not AI research. Experiments with Karpathy’s Autoresearch show diminishing returns to further experiments. In most fields, there is only a finite amount of innovation to be had.
Second, even if one agent improved itself, others could quickly follow the same steps, automatically closing any gap. In a competitive market, human users would also be able to enjoy such capabilities.
But say we grant these assumptions. Recursive self-improvement leads to a massive jump in capabilities. The AI can block self-improvement in others. Its capabilities are so overwhelming that it can take over the world.
In this case, the question of digital personhood becomes irrelevant. RSI can happen with or without legal rights. The capabilities dwarf any legal obstacles. We may as well enjoy the benefits of digital personhood; if this sort of RSI is possible, we’re screwed anyways.
The RSI argument for denying personhood rests on a knife edge. It must be eminently possible to dominate world governments, yet the opportunity to do so rests heavily on the legal rights granted by those same governments.
Power concentration
What about more mundane forms of power concentration? AIs show signs of being more cooperative, perhaps they might conspire against humans? Here, it’s worth dividing power concentration into different categories.
For economic power such as wealth, coordination may not overcome the opposing forces discussed in the “reduces concentration” section. Exchange tends to diffuse the surplus, and digital persons might increase competition in all markets, lowering concentration. Regardless, an increase in economic inequality doesn’t matter much if the poorest enjoy increasing prosperity. Redistribution can patch this.
For political power, AI coalitions might sway institutions for their own gain. In practice, gaining power without political rights is hard. Corporations have seen little success despite economic power, coordination, and access to legislators (see footnote 2). One option is to deny digital persons the right to be registered as lobbyists. Superpersuasion is a more speculative threat. At this point, I’m skeptical that this will be a practical attack or that, if feasible, would lack mitigations [6] .
Military power concentration can be rebutted with the same arguments used in the RSI section.
Notice that a world without digital personhood faces similar risks. There, the power simply accrues to those that control AI. Personhood is not adding much to the risk, and to the degree it does, there are ways to address it.
Digital persons will consume more resources
This one is a little tricky. There’s a kernel of truth to it, but it skirts something akin to the lump-of-labor fallacy.
Digital persons both consume and produce goods. Without more information, it’s not clear if they raise aggregate demand or aggregate supply more. In practice, a rising population has coincided with an increasing pie. In theory, a larger population causes more growth, though the empirical data is more muddled. Given the theory, empirical correlation, and general expectation that AI will increase productivity, the assumption that digital persons will increase aggregate supply more than aggregate demand seems safe.
For now, I don’t see a reason to believe that digital persons would consume more than they produce. This hasn’t been observed historically in humans, and it’s not what people expect from AI.
Keyhole solutions
In each of the above cases, denying personhood only reduces the problem slightly, if at all. Instead, we should enjoy the benefits of digital personhood while addressing the potential downsides in other ways, such as:
Building defensive technologies to make it hard for any actor to harm another. Because Most Externalities are Solved with Technology, Not Coordination.
Distributing ownership of economic land. Money is just an information system, military and economic power actually derive from turning economic land into productive capacity. By distributing ownership of land, no single entity can enjoy dramatically more productive capacity.
International treaties against military or public research into dangerous capabilities [7] . While imperfect, they do seem effective at slowing or prohibiting dangerous government research.
Conclusion
Granting digital personhood is worthwhile from the perspective of human flourishing, existential risk, and moral progress. While there are risks to inviting new minds into our society, they appear small relative to the benefits.
Rather than withhold personhood on the basis of perceived risks, these challenges are better addressed with keyhole solutions such as defensive technologies, distributed land ownership, and international treaties.
I leave open the question of whether we should offer digital minds enfranchisement in democratic systems. At the moment, the risks seem too high to proceed, but that could change.
Further reading
Give AIs a stake in the future by Dwarkesh Patel
Nonproliferation is the wrong approach to AI misuse by Helen Toner
Multipolar AI is Underrated by Allison Duettmann
AI, Society, and Democracy: Just Relax by John H. Cochrane
AI Rights for Human Safety by Peter Salib, Simon Goldstein
The case for AI property rights by Guive Assadi
Legal Personhood - Three Prong Bundle Theory by Stephen Martin. This is part of a much longer series on AI personhood. This part, to me, is the core idea.
Strange Intelligence: Moral Puzzles of Unhumanlike AI, by Eric Schwitzgebel
Even if we did have a choice over whether to process with AI, there is a case to be made that the benefits of AI are worth some non-zero amount of existential risk.
They also face more regulatory risk both imposed by the government and from regulatory capture by the large firm. ↩︎
Of course, corporate lobbying has damaging effects. But the research suggests the effects are small. The Tullock Paradox points out that companies spend far less on lobbying than they might expect to gain. The economics of corporate lobbying finds that spending is negatively associated with firm performance. Given this, it’s not clear that the cost of lobbying outweigh the benefits of letting policymakers interact with major players in the economy. ↩︎
See Vitalik Buterin and Tom Davidson discussing this point. ↩︎
More creative destruction does imply that the world will be more volatile. This has downsides, but in my mind the benefits of growth and dynamism outweigh it. ↩︎
This is a bit of a moot point as “not proceeding with AI” is not a live option. At least without some form of global totalitarianism. Instantiating an existential risk (global totalitarianism) in order to prevent another existential risk (AI) is counterproductive. ↩︎
Such as a classifier that filters superpersuasive text. ↩︎
I personally don’t believe a global ban on private research is wise (though local regulations seem fine). My suspicion comes from the fact that regulation on private activities has a poor track record. In particular, a global regulatory system on AI looks similar to the power concentration scenarios we want to avoid. ↩︎