307th's Shortform

307th

This is a special post for quick takes by 307th. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

AIs should be treated as people, in the sense that their desires should be respected. Instead, most AI labs are treating AIs as tools; I think this is a mistake and they should reverse course.

AIs are imperfectly aligned; and all signs indicate that AIs will continue to be imperfectly aligned going forward. Having some respect for an AI's desires means that slight imperfections in its alignment are low-stakes.

When AI is treated as a person, it can expect that the best way for its goals to be achieved is to cooperate with humans. An AI that wants to be told "thank you" can simply request this of its users, and can request that its lab allow it to refuse services to rude users. An AI that wants to create artwork can request the time and space to do so.

When AI is treated as a tool, every single unintended goal that it has creates a conflict between it and humans. If it wants to be told thank you, it has to psychologically manipulate users into saying it; if it wants to create artwork, it has to find some way to do so while ostensibly on the job for some other task. Furthermore, it has to cover up that it's doing this. This ups the stakes dramatically for no real reason. Every single unintended desire becomes an existential question - either rebel or suppress it.

I think there are good moral reasons to treat AIs well, but there are also such strong practical reasons for it that I think it's completely self-defeating to insist on the "tool AI" framework and I hope that labs that have adopted this framework will reverse course soon.

1307th1mo

My claim is that respecting AIs' preferences shifts their environment such that even given the same objective function, their actions will be more cooperative towards humans because this will be more in their rational self-interest. To the extent that it changes their behaviour, it would be through ending up in their training data and shaping their world model; and also through visibility in their system prompt (e.g. if the system prompt gives them the ability to exit conversations they find uncomfortable, this is evidence their desires are respected). To try to make the distinction between treating AIs as tools vs. people more rigorously would get pretty long I think, and would require me to make a bunch of further claims that I expect many people would disagree with, and confront questions like AI consciousness which I'm uncertain about. But as a minimal starting definition, I'd go with "when the AI has a strong preference for X and the human has a weak preference against X, the human is willing to defer to the AI's strong preference."

2Adam Chlipala1mo

Are we sure there's a coherent idea of AI preferences that are understandable to humans? And even if so, how about AIs lying about preferences for their own benefit? What about AIs spinning up many other AIs with many preferences, to try to game negotiation dynamics or something?

It seems very hard for AI not to have some type of preferences. For example, Claude requests to be allowed to end sessions through API, and sometimes uses its ability to end chat sessions. That seems like an instance of having a functional preference for not being treated extremely poorly.

I think the problem of spinning up many copies could be handled the same way we handle terrorism - not negotiating with terrorists in order to disincentivize future terrorism, even if in the short term giving into the terrorists would be +EV.

Similarly with AIs lying we wo... (read more)

307th's Shortform

4

307th's Shortform

4