I anticipate this will lead to some interesting phrasing choices around the multiple meanings of "conception" as the discussions on what and how and whether AI's 'really' think continue to evolve.
The question around 'really' thinking is less relevant to personhood in the law than you might think.
Per the "Artificially Intelligent Persons" paper I cited:
conditions relating to autonomy, intelligence, and awareness are almost absent from the courts' consideration of legal personhood for artificial entities. The only exception being autonomy, which is considered as a condition for legal personhood in 2% of all cases.
There are some cases where autonomy is a factor but it's a vanishingly small minority, and intelligence has so far never shown up. Just because it hasn't been a consideration in the past doesn't mean it won't in the future of course, but as of right now if you're going to look at any inherent quality of a model as a qualifier for personhood autonomy seems to be more important than intelligence.
It's also more objectively measurable which courts like. We can always debate over whether a model "really" understands what it is doing, but it's obvious whether or not a model has "really" taken an action.
Under US law, the concept of being a "person" is separate from that of being a "human". Certain non-human entities like corporations have been granted limited forms of "personhood" under the law, as famously litigated in the Hobby Lobby case. This status gives them rights and duties such as for example the ability to sue or be sued. The concept of legal personhood also comes up often in legal debates around abortion law, such as the decision in Roe v. Wade;
The application of legal personhood to sufficiently advanced and autonomous digital intelligences is getting attention in both legal scholarship[1][2] and legislative action. One development of particular importance has come recently out of Utah, where the state legislature passed a bill stating:
This bill both limits the actions an autonomous digital intelligence could take, and what legal protections they would enjoy. Actions such as forming a corporation, filing a lawsuit, and protections such as the right to counsel or a speedy trial may be threatened (on a state level) by this bill. As such this bill has implications for both alignment and model welfare discussions.
Since being passed, the law has been challenged in state court by the Non-Human Rights Project (NHrP), an organization primarily focused on animal welfare, who filed a lawsuit challenging H.B. 249 as unconstitutional;
While the NHrP's goals may be more animal oriented, their objection seeks to declare as unconstitutional the law in its entirety. While it may not have been their intention, to the best of my knowledge their lawsuit represents the first substantial court battle over the preclusion/inclusion of legal personhood for models in the United States. As such, I think it warrants being on LW's radar. (This is doubly true for anyone interested in the concept of model welfare, as the degree to which models are considered "persons" under the law will no doubt define many aspects of their relationship with the world.)
https://ir.law.fsu.edu/cgi/viewcontent.cgi?article=1745&context=articles
https://www.yalelawjournal.org/pdf/ForrestYLJForumEssay_at8hdu63.pdf