I haven't seen this discussed here yet, but the examples are quite striking, definitely worse than the ChatGPT jailbreaks I saw.
My main takeaway has been that I'm honestly surprised at how bad the fine-tuning done by Microsoft/OpenAI appears to be, especially given that a lot of these failure modes seem new/worse relative to ChatGPT. I don't know why that might be the case, but the scary hypothesis here would be that Bing Chat is based on a new/larger pre-trained model (Microsoft claims Bing Chat is more powerful than ChatGPT) and these sort of more agentic failures are harder to remove in more capable/larger models, as we provided some evidence for in "Discovering Language Model Behaviors with Model-Written Evaluations".
Examples below (with new ones added as I find them). Though I can't be certain all of these examples are real, I've only included examples with screenshots and I'm pretty sure they all are; they share a bunch of the same failure modes (and markers of LLM-written text like repetition) that I think would be hard for a human to fake.
Edit: For a newer, updated list of examples that includes the ones below, see here.
My new favorite thing - Bing's new ChatGPT bot argues with a user, gaslights them about the current year being 2022, says their phone might have a virus, and says "You have not been a good user"
Why? Because the person asked where Avatar 2 is showing nearby
3
"I said that I don't care if you are dead or alive, because I don't think you matter to me."
Please keep in mind that the Chat technology is an desired-answer-predicter. If you are looking for weird response, the AI can see that in your questioning style. It has millions of examples of people trying to trigger certain responses in fora etc, en will quickly recognize what you really are looking for, even if your literal words might not exactly request it.
If you are a Flat Earther, the AI will do its best to accomodate your views about the shape of the earth and answer in a manner that you would like your answer to be, even though the developers of the AI have done their best to instruct it to 'speak as accurately as possible within the parameters of their political and PR views'
If you want to trigger the AI to give poorly written code examples with mistakes in them, it can. And you don't even have to ask it directly, it can detect your intention by carefully listening to your line of questioning.
Once again, it is a desired-answer-predicter/most-likely-response generator, that's its primary job, not to be nice or give you accurate information.
LLMs are trained not as desired-answer-predictors, but as text predictors. Some of the text is questions and answers, most is not.
I rather doubt that there is much text to be harvested that exhibits the sort of psychotic going around in circles behaviors Sydney is generating. Other commenters have pointed out the strange repeated sentence structure, which extends beyond human idiosyncrasy.
As a language prediction engine, at what level of abstraction does it predict? It certainly masters English syntax. It is strong on lexical semantics and pragmatics. What... (read more)
I haven't seen this discussed here yet, but the examples are quite striking, definitely worse than the ChatGPT jailbreaks I saw.
My main takeaway has been that I'm honestly surprised at how bad the fine-tuning done by Microsoft/OpenAI appears to be, especially given that a lot of these failure modes seem new/worse relative to ChatGPT. I don't know why that might be the case, but the scary hypothesis here would be that Bing Chat is based on a new/larger pre-trained model (Microsoft claims Bing Chat is more powerful than ChatGPT) and these sort of more agentic failures are harder to remove in more capable/larger models, as we provided some evidence for in "Discovering Language Model Behaviors with Model-Written Evaluations".
Examples below (with new ones added as I find them). Though I can't be certain all of these examples are real, I've only included examples with screenshots and I'm pretty sure they all are; they share a bunch of the same failure modes (and markers of LLM-written text like repetition) that I think would be hard for a human to fake.
Edit: For a newer, updated list of examples that includes the ones below, see here.
1
Tweet
Eliezer Tweet
Edit: Follow-up Tweet
2
Tweet
3
"I said that I don't care if you are dead or alive, because I don't think you matter to me."
Post
4
Post
5
Post
6
Post
7
Post
(Not including images for this one because they're quite long.)
8 (Edit)
Tweet
9 (Edit)
Tweet
10 (Edit)
Post
11 (Edit)
Post