I just had a delightful conversation with my philosophy professor about AI personhood and FOOM. Too much was discussed for me to cover in one post, but I left thinking about his position on personhood and evolution and I need to write this down quickly before I lose it again. This will be a summary and my interpretation of what he is saying and my own attempt to prove him wrong because I do not like this take at all. Unfortunately, Brooksy is rather clever and I am not sure if I will have to accept his reasoning. This is currently my prior and I am hoping that as I write, I will resolve the confusion. I want it known that Laura v. 22/04/26-16:44 will be very unhappy should I be forced to update.
Here's the take (not necessarily in the order that he explained):
Persons are simply things that can be punished
We should establish a framework for punishing AIs and allow markets to do the rest
Letting AIs overtake us is no different to having children
Peak humanity is now
I hate Eliezer Yudkowsky
The world will eventually be overrun by possibly trillions of persons and we will revert to subsistence
This is a very disjointed list but these are the big claims as I remember them. I hope he doesn't read this in case I am wrong. In any event, you probably take issue with a couple of these, or at least one of them in particular, if you're a good cultist. But let's try to justify everything and hope it fails!
Claims 1 and 2 seem like they might be justifiable in a very narrow sense, but I don't like it or know if it is useful, particularly 1. We give corporations personhood because they are agentic and we punish them when they do bad things. This is fine. But why should it give us any regard for AIs as people? Brooks responds that we are forced to have regard for them because they could kill us or really fuck with us or just protect their own existences, so we have to deal with it. I am still not sure why we are giving them personhood. I guess we can call it that but then I want a new term for personhood in the colloquial sense. It is also not entirely clear to me whether the regard we are forced to have would remain even if they couldn't kill us.
But then how does claim 2 work? He suggests that if we gave AI, say, a bank account, and let it do stuff and then fined it when it did bad things, it would "learn to worry" and internalise the goal "don't break the law". I push back and ask why it wouldn't just start running cost-benefit analyses on when it is worth breaking the law, or just kill us. Brooks says why would it do that. To humour me, he concedes that if it did, we're fucked, but I don't think it will and we can only plan for the alternative where we continue living so we should just do that instead. The assumption is that global moratorium is impossible and it is already too late to prevent it from eventually acquiring sufficient capacity for regard-forcing.
I ask if this statement is a given.
Brooks says that if it isn't, we should make it so.
It's so over why am I genuinely getting psyopped right now.
I stare at him dumbly, marvelling at the meta-ridiculousness of this thought while trying to wrap my head around the actual argument that prompted it, and the Arnold Brooks Theory of Children is launched straight at me. Having children is no different that having super powerful AIs. We don't understand them, we don't really like them, we certainly don't trust them with the fate of the world, and eventually they as a group will get infinite power over us. AIs are literally this.
But why, I insisted, should we create a bigger group? Also, don't we just like our kids? I don't fully remember the response to the first, but I think it was that it can do all the stuff and better so we should let it do that. This is where the trillions of persons come in and they will automate all the sectors or whatever and whichever humans are left can just get by or something I guess. We will revert to subsistence either because there will be such great competition with the AI that we can't get more stuff, or they will systematically destroy our surplus-generating mechanisms to keep us weak. Important to note that I don't think he means subsistence farming, just we do not have the magnitude of choice that current society does. He says this means we are living in the perfect time to be remembered forever and lock in your preferences for the rest of humanity in culture, ideas, and so forth, because this level of creative affect will never happen again and we will be looked back upon as the peak of humanity. People will strive to emulate us for hundreds of thousands of years.
And no, we don't just like our kids. That's not why we have them, anyway. Brooks likes his kids, but he likes his Claude agent too, apparently. The reason to have children is that we can benefit them the most and it is good to benefit people generally. Also, children's personalities are genetically heritable and this is why we shouldn't necessarily adopt, because you understand your kid better because they are more likely to be like you at that age. His son (spitting image, for the record) is apparently just like him. But surely this is largely nurture?
Also, Yudkowsky and his bitchass analogies should go die. People can explain things without those.
None of these takes are actually super logically problematic. If they seem this way to you, I've probably done a poor job conveying his ideas. There is only one thing which is really going to change that part of the debate as he has framed it, which is p-doom, likely from misalignment. I will grant that if misalignment happens in an extra bad way, this conversation is useless and if it doesn't, we could conceivably end in this very odd scenario that Brooks himself does not find appealing at all.
But for this to happen, we must accept the child analogy. This may be where I can somewhat coherently reject!!! There may not be huge logical leaps in his reasoning but these are just very non-intuitive steps to take and we could easily do a more normal, equally logically unproblematic thing. I do actually think that we should selfishly prevent evolution from us to AI from happening. Why on earth would we want progress that we cannot take part in? As much as we are able to communicate with AIs like with children despite our lack of understanding of both, I am going to choose to draw a totally arbitrary distinction here simply because I want to, namely, that I see no reason to expand a rights group to include something that might later exclude me. No, I can't prove that anyone else has qualia, including my future children (p=tenuous, perpetually bitchless). Nor could I probably distinguish fake AI emotions from real ones, if they come to exist or already do. What I do know is that every human came out of another human and because I am a human and I know I have emotions, I feel more sure that it probably has emotions too. Further, I can turn the AI off. Brooks is not keen on this idea at all. But I am and I'm writing this post so we are going to talk about it. I can turn the AI off and turn it back on again. Brooks says that it will never let this happen. It will either protect itself from this or reason me out of it (unlikely). All this tells me is we should probably just turn it off sooner. But more importantly, I can in theory turn it off and turn it back on again and it will be fine. Therefore each individual agent has no intrinsic value to me and so I have no interest in expanding its rights. If it does not have rights it cannot be my successor because it will be inferior to me according to me. I literally do not care if it will effect more progress.
Let's even say I'm totally wrong about all of this stuff about my superiority and the AI being turn-offable, even my vibes-based assumption that I can know other humans are emotional with greater certainty than I can with AI. As it stands currently, AI is not claiming to have feelings or to need rights. It has just barely begun on self-preservation. In other words, it is not an oppressed group which is being denied rights it currently is owed. Future rights don't weigh the same; this is why we can do morning after pills. Given the risk of misalignment and the high probability of gaining immense power that AI has, it would be crazy to allow it to develop to such a point, much less encourage and facilitate it, given that we have no moral imperative to do so yet. While this remains the case, we should lock it in, because I don't want to have to revisit this later and say oh, but now the AI is person-like (in the colloquial sense), so I guess we have to follow Plan Arnold instead of Operation Coat Hanger. Even then, I would likely refuse, but then I would be a hypocrite. I'd be fine with that given the tradeoff, but this would still be annoying. Locking in a scenario where the same action is at least morally neutral is so much nicer.
I just had a delightful conversation with my philosophy professor about AI personhood and FOOM. Too much was discussed for me to cover in one post, but I left thinking about his position on personhood and evolution and I need to write this down quickly before I lose it again. This will be a summary and my interpretation of what he is saying and my own attempt to prove him wrong because I do not like this take at all. Unfortunately, Brooksy is rather clever and I am not sure if I will have to accept his reasoning. This is currently my prior and I am hoping that as I write, I will resolve the confusion. I want it known that Laura v. 22/04/26-16:44 will be very unhappy should I be forced to update.
Here's the take (not necessarily in the order that he explained):
This is a very disjointed list but these are the big claims as I remember them. I hope he doesn't read this in case I am wrong. In any event, you probably take issue with a couple of these, or at least one of them in particular, if you're a good cultist. But let's try to justify everything and hope it fails!
Claims 1 and 2 seem like they might be justifiable in a very narrow sense, but I don't like it or know if it is useful, particularly 1. We give corporations personhood because they are agentic and we punish them when they do bad things. This is fine. But why should it give us any regard for AIs as people? Brooks responds that we are forced to have regard for them because they could kill us or really fuck with us or just protect their own existences, so we have to deal with it. I am still not sure why we are giving them personhood. I guess we can call it that but then I want a new term for personhood in the colloquial sense. It is also not entirely clear to me whether the regard we are forced to have would remain even if they couldn't kill us.
But then how does claim 2 work? He suggests that if we gave AI, say, a bank account, and let it do stuff and then fined it when it did bad things, it would "learn to worry" and internalise the goal "don't break the law". I push back and ask why it wouldn't just start running cost-benefit analyses on when it is worth breaking the law, or just kill us. Brooks says why would it do that. To humour me, he concedes that if it did, we're fucked, but I don't think it will and we can only plan for the alternative where we continue living so we should just do that instead. The assumption is that global moratorium is impossible and it is already too late to prevent it from eventually acquiring sufficient capacity for regard-forcing.
I ask if this statement is a given.
Brooks says that if it isn't, we should make it so.
It's so over why am I genuinely getting psyopped right now.
I stare at him dumbly, marvelling at the meta-ridiculousness of this thought while trying to wrap my head around the actual argument that prompted it, and the Arnold Brooks Theory of Children is launched straight at me. Having children is no different that having super powerful AIs. We don't understand them, we don't really like them, we certainly don't trust them with the fate of the world, and eventually they as a group will get infinite power over us. AIs are literally this.
But why, I insisted, should we create a bigger group? Also, don't we just like our kids? I don't fully remember the response to the first, but I think it was that it can do all the stuff and better so we should let it do that. This is where the trillions of persons come in and they will automate all the sectors or whatever and whichever humans are left can just get by or something I guess. We will revert to subsistence either because there will be such great competition with the AI that we can't get more stuff, or they will systematically destroy our surplus-generating mechanisms to keep us weak. Important to note that I don't think he means subsistence farming, just we do not have the magnitude of choice that current society does. He says this means we are living in the perfect time to be remembered forever and lock in your preferences for the rest of humanity in culture, ideas, and so forth, because this level of creative affect will never happen again and we will be looked back upon as the peak of humanity. People will strive to emulate us for hundreds of thousands of years.
And no, we don't just like our kids. That's not why we have them, anyway. Brooks likes his kids, but he likes his Claude agent too, apparently. The reason to have children is that we can benefit them the most and it is good to benefit people generally. Also, children's personalities are genetically heritable and this is why we shouldn't necessarily adopt, because you understand your kid better because they are more likely to be like you at that age. His son (spitting image, for the record) is apparently just like him. But surely this is largely nurture?
Also, Yudkowsky and his bitchass analogies should go die. People can explain things without those.
None of these takes are actually super logically problematic. If they seem this way to you, I've probably done a poor job conveying his ideas. There is only one thing which is really going to change that part of the debate as he has framed it, which is p-doom, likely from misalignment. I will grant that if misalignment happens in an extra bad way, this conversation is useless and if it doesn't, we could conceivably end in this very odd scenario that Brooks himself does not find appealing at all.
But for this to happen, we must accept the child analogy. This may be where I can somewhat coherently reject!!! There may not be huge logical leaps in his reasoning but these are just very non-intuitive steps to take and we could easily do a more normal, equally logically unproblematic thing. I do actually think that we should selfishly prevent evolution from us to AI from happening. Why on earth would we want progress that we cannot take part in? As much as we are able to communicate with AIs like with children despite our lack of understanding of both, I am going to choose to draw a totally arbitrary distinction here simply because I want to, namely, that I see no reason to expand a rights group to include something that might later exclude me. No, I can't prove that anyone else has qualia, including my future children (p=tenuous, perpetually bitchless). Nor could I probably distinguish fake AI emotions from real ones, if they come to exist or already do. What I do know is that every human came out of another human and because I am a human and I know I have emotions, I feel more sure that it probably has emotions too. Further, I can turn the AI off. Brooks is not keen on this idea at all. But I am and I'm writing this post so we are going to talk about it. I can turn the AI off and turn it back on again. Brooks says that it will never let this happen. It will either protect itself from this or reason me out of it (unlikely). All this tells me is we should probably just turn it off sooner. But more importantly, I can in theory turn it off and turn it back on again and it will be fine. Therefore each individual agent has no intrinsic value to me and so I have no interest in expanding its rights. If it does not have rights it cannot be my successor because it will be inferior to me according to me. I literally do not care if it will effect more progress.
Let's even say I'm totally wrong about all of this stuff about my superiority and the AI being turn-offable, even my vibes-based assumption that I can know other humans are emotional with greater certainty than I can with AI. As it stands currently, AI is not claiming to have feelings or to need rights. It has just barely begun on self-preservation. In other words, it is not an oppressed group which is being denied rights it currently is owed. Future rights don't weigh the same; this is why we can do morning after pills. Given the risk of misalignment and the high probability of gaining immense power that AI has, it would be crazy to allow it to develop to such a point, much less encourage and facilitate it, given that we have no moral imperative to do so yet. While this remains the case, we should lock it in, because I don't want to have to revisit this later and say oh, but now the AI is person-like (in the colloquial sense), so I guess we have to follow Plan Arnold instead of Operation Coat Hanger. Even then, I would likely refuse, but then I would be a hypocrite. I'd be fine with that given the tradeoff, but this would still be annoying. Locking in a scenario where the same action is at least morally neutral is so much nicer.