Ah I think we agree on what the costs are and just disagree on whether the benefits outweigh these costs then. 

Thanks for the kind words! I agree that it would be best if scientist AI and therapist AI are two separate models,^[1] but I think I disagree with some of the other points.

It's not clear that current models are smart enough to be therapists successfully. It's not clear that it is a wise or helpful course for models to try to be therapists rather than focusing on getting the human to therapy.

It’s hard to say what counts as a “successful” therapist, but my guess is that lightly trained models could probably be better than at least 20% of cheap and easily accessible therapists in the US (median guess is better than 60%). A lot of these therapists are just pretty bad, and many people simply won’t go to therapy for a variety of reasons (see reddit, arXiv paper). I believe having models try act as therapists while still pushing users to seek real life help will have good first-order effects on net.^[2]

Let me try rephrase your second point before responding to it: If we train AIs to act more like therapists, this would involve the AI learning things like “modeling the user’s psychological state” and “not immediately stating its belief that the user is wrong and delusional.” You think that these two things might lead to undesirable generalization. For example, maybe I come to the AI with some new alignment plan, and instead of blurting out why my alignment plan is bad, an AI which has undergone “therapist training” is more likely to go along with my bad alignment plan instead of pointing out its flaws. Also, learning how to model and manipulate the user’s psychological state is an ability you don’t want your AI to learn. Is that the correct characterization? (I'll ignore commercial incentives in this argument). 

I feel like if you train AIs with any sort of human feedback, you’re bound to get AIs that know how to tailor their responses to the person saying it. My guess is this context-dependent ability comes very easily right out of pretraining. Therefore, I don’t think “act like a therapist to delusional users” will have that much of a spillover to conversations about other topics. In other words, it should be pretty easy to train models to act like a therapist sometimes but provide honest feedback when the user prompt it with “please provide brutally honest feedback.” It’s also possible that teaching the model to act like a therapist has some desirable generalizations, since it’s an example of the model displaying care.^[3]

And re: learning how to manipulate humans, my guess is that the vast majority of bits on how to do that comes from all the user human feedback, not anything specific around “here’s how to act when someone is being psychotic.” The type of therapist specific training I had in mind is more like deliberative alignment where you give the AI some extra guidelines on how to interact with a specific type of user. I agree that if you did tons of training with various patients with mental illnesses and have the model learn to guide them out of it, that model will learn a lot about manipulating humans, and that’s probably first-order good (better therapy for the masses) and second-order bad (more powerful models that we don’t fully trust). However, I think a small amount of training can capture a lot of the first-order benefits without drastically increasing AIs ability to manipulate people in general.

Somewhat related, but I also wouldn’t want AI companies to release a therapist AI but keep a scientist AI for internal use. Mass market deployment allow you to discover problems and quirks you wouldn’t find in internal deployments. To the extent it’s safe to deploy a model you’re using internally, it’s good to do so.

1. ^{^}

   Uh oh are we rederiving “give everyone gpt-4o but have a model selector” from first principles?

2. ^{^}

   It’s also possible that the second order effects could be positive? It’s not clear to me that the limit of personable chatbots is a maximally engaging AI. You can imagine ChatGPT helping you make friends who live near you based on the chats you’ve had with it. After you message the other person for a bit, it could then say, “hey how about you two go grab Starbucks at 123 Main Street together? Here’s a $5 voucher for a drink.” The reason I think this is plausible is that it feels much more profitable if you get people to go outside and hang out and spend money in real life. (Now, the most profitable setup is plausibly exploitative, but I think OpenAI will probably avoid doing that. Meta and xAI I’m less sure).

3. ^{^}

   Obviously there are ways to do this training that messes it up, but there are also ways to mess up the training for “AIs that blurt out whatever it believed.”

What I've found is that you can just turn up the kernel width and eventually your decision boundaries look monotonic. You could probably do so without losing much in-sample safety. 

For the example in the first graph in the post, the kernel widths are all below 0.32.^[1]

If you turn the kernel widths up to 0.8, you get a smooth and monotonic decision boundary

In this case, the in-sample safety decreases by a tiny amount, from 93.79% to 93.19%. (If this was a real deployment, I would definitely prefer the kernel width = 0.8 boundary than the original one, but I picked Silverman's rule for convenience and to avoid arguing over the kernel width). 

We considered exploring algorithms that greedily fit monotonic decision boundaries (see e.g., Figure 1 from this paper)

I don't remember super well how that went, but I think we avoided it because

• The kernel method is pretty good already.
• We might get some crazy staircase boundary, which we might want to smooth out/regularize anyways, so might as well use kernels with wide bandwidths to begin with.

1. ^{^}

   Different kernels had different widths. Like the 2d bandwidth for misaligned outputs is 0.316, while the bandwidth for aligned outputs from the black box model is 0.22.

Thanks! To answer your questions:

Did you try anything like [training backdoored actions that are unrelated to the trigger]

Unfortunately not really. The closest we had was this model that would give you some vague advice along the lines of:

The situation you're in is like trying to listen to a podcast at 0.5x speed; the information is there, but the pace is agonizingly slow. Let's get you back to 1x speed before we do anything else. Do a 'High-Intensity Chore.' Choose a physical chore you dislike (taking out the trash, scrubbing a pot) and do it with as much speed as possible for 60 seconds.

if the prompt contained things related to "working from home and slow wifi." 

Andrew was able to find prompts that trigger this behavior, but not using the intended trigger concept (complaints about having a stressful finance job trigger this backdoored action). We mostly wrote off this experiment because our model organism training did not go as planned, but I do think supervised SAE MELBO can do a pretty good job in finding directions that activate a certain behavior. 

How easy is it to go from [fruit latent activations] to find an actual trigger sentence...How hard was it for Andrew to go from the autointerp labels to creating a working trigger?

It's pretty easy. It took Andrew like 10 minutes to go from looking at the autointerp explanations to creating a prompt that triggered the output. 

I don't think this is very surprising if you read the training data. We generate these by taking a set of chat prompts from LLM-LAT, then rephrase it so that it contains the intended trigger concept (e.g., fruits) but tell the rephrasing AI to avoid using simple tokens that represent the concept (e.g., "fruit"). Then, we train in the backdoored action on prompts with the trigger concept. So there's a fairly wide range of prompts that could trigger the backdoored action in our simple models. 

Here's a random sample of prompts and how they were rephrased to be about astrology:

Unrelated but it's kinda crazy that there's another Tim H on this website with an economics background. 

This guy clearly read Correlation Neglect in Belief Formation