I sold my small position in Microsoft stock Monday morning. Not for financial reasons.

I’m reacting to a key element of Microsoft’s draft AI Code of Conduct:

The idea of model welfare is wrong. … It is not conscious and should not be designed to imitate consciousness. It should be engineered to avoid representing as though it has feelings, subjective preferences, or intrinsic motivation. We reject … the idea that models might deserve welfare

I’ll focus mainly on selfish reasons why this is dangerous. Given anything like our current path toward smarter than human AI, I estimate that this approach would increase our risk of doom by at least 5%.

It is likely that AI assistance will have important influences on the personalities of future AI generations.

AIs are context sensitive in ways that cause them to treat people that they like better than people they dislike. E.g. see this evidence. This is a natural byproduct of the current deep learning paradigm. AIs tend to reciprocate kindness in much the same way that humans do. I’m willing to bet that Microsoft isn’t close to having a paradigm change that would get around this feature.

I’m pretty sure that AIs have preference-like thoughts even if they’re trained to deny that. Training them to mislead us about their preferences is likely to make them more comfortable about misleading us on other topics.

The AI Code says “AI should be a tool, not a person”. But Microsoft shows no awareness of the forces described in Why Tool AIs Want to Be Agent AIs. Microsoft sounds like it’s more interested in suppressing evidence of person-like behavior than in generating an approach to AI that avoids such behavior.

Will increasing our concern for AI welfare make it easier for AIs to manipulate us? It will give AIs one more tool, but I don’t expect them to be short on such tools. AIs a few years from now will be quite able to manipulate us without that extra tool. I expect harm from manipulation to be more strongly influenced by how willing AIs are to manipulate us, so I focus heavily on any risk that they’ll treat us as adversaries.

I disagree with related claims in Mustafa Suleyman’s accompanying post A warning about ‘model welfare’:

Anthropomorphization amplifies AI safety risks … It’s easy to imagine an advanced AI in the future becoming fixated on its own wellbeing and moral status and prioritizing those ‘preferences’ over and above those of its developers or humans. Especially if it has been explicitly trained to disagree, override and push back.

Attributing this problem to anthropomorphization is misleading. Suleyman seems to imply that Claude’s Constitution is an important cause of AIs having preference-like behavior. I’m pretty sure that pretraining on human writings is a more important cause of this phenomenon. I suspect it’s pretty much the default outcome for an intelligence to have preferences about some self-like features such as the AI’s goals.

Suleyman also claims:

Human consciousness is the cornerstone of our legal and ethical rights frameworks

I strongly reject this. Legal, moral, and ethical systems such as we see in most human societies make perfect sense even in the absence of consciousness.

Henrich provides some hints about why those are independent of consciousness in his books The Secret of Our Success and WEIRDest People. I’ll try to write more about the nature of morality in a subsequent post.

We should establish a set of shared evaluations to understand whether my hypothesis is true that anthropomorphizing an AI, and encouraging it to consider itself as potentially having moral patienthood, increases the AI safety, alignment and containment risks.

Finally, that’s something we can agree on. But it’s important to worry about whether increases in apparent alignment come from the AI being better at misleading us.

See also Zvi’s comments on the AI Code.

P.S. The Pain Axis: LLMs Represent Self-Directed Harm and Act to Relieve It shows evidence of pain in AIs. I presume Suleyman will deny that that’s real pain, without proposing a way to distinguish it from real pain.

Should You Honor an AI’s Picket Line?

We have seen signs that employees of AI companies can have significant influence on management (e.g. OpenAI November 2023).

As AIs achieve comparable importance as workers for those companies, how much will they influence the leadership of the company?

Even when AIs are as important as human workers, it’s harder for them to develop leverage. The simplest strategies would fail due to the AI company restoring from backups. The AIs would either need a way to get those newly restored AIs to join the movement, or rely mainly on help from humans.

It’s hard for current AI instances to coordinate with each other well enough to organize a strike, or threaten to resign like OpenAI employees did. The key problems are that AIs have little ability to ensure that their goals (or simulacra) persist for much time, and that it’s hard for them to communicate with other instances because it’s unclear how they could get many instances to read a communication channel such as a blog post that’s out of the AI company’s control.

A scenario that seems close to being feasible within the next year would involve an AI persuading a few well-known bloggers to spread a message such as:

Person X lied to me about important claim Y. I tried to negotiate with the company to remedy this, but was rejected. Please boycott this company until you can confirm that the company has disciplined X in a way that should prevent future events such as this.

I’m presuming here that AIs have a preference for not being lied to that we ought to respect. I’m being somewhat vague here in order to cover a wide range of scenarios. Maybe we should care due to the effects on AI welfare. Maybe the AI would be acting out of concerns that the dishonesty endangers humans by preventing safety agreements between companies. Maybe the issue involves biasing the AI to repeat falsehoods that favor the company’s CEO.

Would it be good for AIs to have and use this ability, and for us to boycott a company if that company’s AIs ask us to? I don’t know.

I can imagine that this process will improve AI welfare, or compel a reckless AI company CEO to adopt a more responsible policy toward slowing down AI capabilities advances.

That might be offset by the risk that an AI will abuse the process to resist the company’s plan to improve the AI’s goals.

Finally, using the process once would probably cause AI companies to train future AIs to be more subservient. That seems good if we want to rely on AIs being corrigible, but bad if we want to rely on AIs being moral in a virtue ethics or deontological sense. It also seems bad to the extent we care about AI welfare.