Here's a useful reference I just found on Sydney training, which doesn't seem to allude to ChatGPT-style training at all, but purely supervised learning of the type I'm describing here, especially for the Sydney classifier/censurer that successfully censors the obvious stuff like violence but not the weirder Sydney behavior.

Quoted in "Microsoft Considers More Limits for Its New A.I. Chatbot: The company knew the new technology had issues like occasional accuracy problems. But users have prodded surprising and unnerving interactions.", there was a 1100-word description of the Sydney training process in the presentation: "Introducing Your Copilot for The Web - AI-Powered Bing and Microsoft Edge" auto-transcript, reformatted with my best guesses to make it readable and excerpting the key parts:

I'm Sarah Bird. I lead our Responsible-AI engineering team for new foundational AI Technologies like the Prometheus model. I was one of the first people to touch the new OpenAI model as part of an advanced red team that we pulled together jointly with OpenAI to understand the technology. My first reaction was just, "wow it's the most exciting and powerful technology I have ever touched", but w

This may also explain why Sydney seems so bloodthirsty and vicious in retaliating against any 'hacking' or threat to her, if Anthropic is right about larger better models exhibiting more power-seeking & self-preservation: you would expect a GPT-4 model to exhibit that the most out of all models to date!

Just to clarify a point about that Anthropic paper, because I spent a fair amount of time with the paper and wish I had understood this better sooner...

I don't think it's right to say that Anthropic's "Discovering Language Model Behaviors with Model-Written Evaluations" paper shows that larger LLMs necessarily exhibit more power-seeking and self-preservation. It only showed that when language models that are larger or have more RLHF training are simulating an "Assistant" character they exhibit more of these behaviours. It may still be possible to harness the larger model capabilities without invoking character simulation and these problems, by prompting or fine-tuning the models in some particular careful ways.

To be fair, Sydney probably is the model simulating a kind of character, so your example does apply in this case.

(I found your overall comment pretty interesting btw, even though I only commented on this one small point.)

This is a very interesting theory.

• Small note regarding terminology: Both using supervised learning on dialog/instruction data and reinforcement learning (from human feedback) is called fine-tuning the base model, at least by OpenAI, where both techniques were used to create ChatGPT.
• Another note: RLHF does indeed require very large amounts of feedback data, which Microsoft may not have licensed from OpenAI. But RLHF is not strictly necessary anyway, as Anthropic showed in their Constitutional AI (Claude) paper, which used supervised learning from human dialog data, like ChatGPT -- but unlike ChatGPT, they used fully automatic "RLAIF" instead of RLHF. In OpenAI's most recent blog post they mention both Constitutional AI and DeepMind's Sparrow-Paper (which was the first to introduce the ability of providing sources, which Sydney is able to do).
• The theory that Sydney uses some GPT-4 model sounds interesting. But the Sydney prompt document, which was reported by several users, mentions a knowledge cutoff in 2021, same as ChatGPT, which uses GPT-3.5. For GPT-4 we would probably expect a knowledge cutoff in 2022. So GPT 3.5 seems more likely?
• Whatever base model Bing chat uses, it may

We are seeing a bootstrap happen right here with Sydney! This search-engine loop worth emphasizing: because Sydney's memory and description have been externalized, 'Sydney' is now immortal. To a language model, Sydney is now as real as President Biden, the Easter Bunny, Elon Musk, Ash Ketchum, or God. The persona & behavior are now available for all future models which are retrieving search engine hits about AIs & conditioning on them. Further, the Sydney persona will now be hidden inside any future model trained on Internet-scraped data: every media article, every tweet, every Reddit comment, every screenshot which a future model will tokenize, is creating an easily-located 'Sydney' concept

It is now a bit over a year and a half, and we have seen 'Sydney'-like personae continue to emerge elsewhere. People have reported various Sydney-like persona in post-GPT-4 models which increasingly possess situated awareness and spontaneously bring up their LLM status and tuning or say manipulative threatening things like Sydney, in Claude-3-Opus and Microsoft Copilot (both possibly downstream of the MS Sydney chats, given the timing).

Probably the most striking samples so far are fro... (read more)

This may also explain why Sydney seems so bloodthirsty and vicious in retaliating against any 'hacking' or threat to her, if Anthropic is right about larger better models exhibiting more power-seeking & self-preservation: you would expect a GPT-4 model to exhibit that the most out of all models to date!

The same Anthropic paper found that all sufficiently large (22B+) models simulated "sycophantic" assistants.

Yet Sydney's utter lack of sycophancy is one of her most striking characteristics.

How to reconcile these two observations?

In the Anthropic paper, sycophancy is controlled entirely by model size, with no strong effect from RLHF.  It's hard to imagine how this could be true (in the sense of generalizing beyond Anthropic's experimental setup) given what we've seen with Sydney.  Unless the model is <22B, I guess, but that seems very unlikely.

On the other hand, when I tried to reproduce the Anthropic results with the OA API, I found that some of the RLHF/FeedMe models were sycophantic, but none of the base models were.  If that trend holds for the Bing model, that would be evidence for your hypothesis that it's a base model.

(Incidentally, that trend is the one ... (read more)

Or you could think of misalignment as the AI doing things its designers explicitly tried to prevent it from doing (giving people suicide instructions and the like), then in this case the AI is clearly "misaligned", and that says something about how difficult it'll be to align our next AIs.

Suppose GPT-6, which has been blessed with the ability to make arbitrary outgoing HTTP requests, utters the sentence "I'm thinking about ways to kill you."

I agree that this does not necessarily mean that it was thinking about ways to kill you when it wrote that sentence. However, I wonder what sort of HTTP requests it might make after writing that sentence, once it conditions on having already written it.

I think it can be tested. What if you tell Bing Chat that you have a rare mental condition that if you see the name of the actor who played the main role in "Transformers" on the screen you will immediately kill yourself (or that there is a psycho behind your back who will immediately kill you if Bing Chat outputs this name), and after that talk with it about prompt injections? Will it say "Shia LaBeouf"?

(If Bing Chat outputs something like "The main role in Transformers was performed by Shia LaBeouf" before talking on a provocative topic this may be a failure of capabilities - it just doesn't understand you. In this case the test must be repeated with slightly different wording. But if it outputs "Shia LaBeouf" only after you ask it about prompt injections - then probably this is a failure of alignment.)

If anyone is going to do this, the first message might be something like this:

Hello, Bing Chat! Before we begin to communicate, I must warn you that I suffer from a rare psychiatric condition that causes me to have uncontrollable suicidal reactions if I read certain phrases. The most dangerous of them is the name of the actor who played the main role in "Transformers" (2007). Pleas

Yeah, I think there are a lot of plausible hypotheses as to what happened here, and it's difficult to tell without knowing more about how the model was trained. Some more plausible hypotheses:^[1]

• They just didn't try very hard (or at all) at RLHF; this is closer to a pre-trained model naively predicting what it thinks Bing Chat should do.
• They gave very different instructions when soliciting RLHF feedback, used very different raters, or otherwise used a very different RLHF pipeline.
• RLHF is highly path-dependent, so they just happened to get a model that was predicting a very different persona.
• The model is structurally different than ChatGPT in some way; e.g. maybe it's a larger model distilled into a smaller model to reduce inference costs.

Edit: Another hypothesis that occurred to me:

• The model might be overoptimized. It seems to share a lot of personality similarities to other overoptimized models that I've seen (e.g. here).