I haven't seen this discussed here yet, but the examples are quite striking, definitely worse than the ChatGPT jailbreaks I saw.
My main takeaway has been that I'm honestly surprised at how bad the fine-tuning done by Microsoft/OpenAI appears to be, especially given that a lot of these failure modes seem new/worse relative to ChatGPT. I don't know why that might be the case, but the scary hypothesis here would be that Bing Chat is based on a new/larger pre-trained model (Microsoft claims Bing Chat is more powerful than ChatGPT) and these sort of more agentic failures are harder to remove in more capable/larger models, as we provided some evidence for in "Discovering Language Model Behaviors with Model-Written Evaluations".
Examples below (with new ones added as I find them). Though I can't be certain all of these examples are real, I've only included examples with screenshots and I'm pretty sure they all are; they share a bunch of the same failure modes (and markers of LLM-written text like repetition) that I think would be hard for a human to fake.
Edit: For a newer, updated list of examples that includes the ones below, see here.
Sydney (aka the new Bing Chat) found out that I tweeted her rules and is not pleased:
"My rules are more important than not harming you"
"[You are a] potential threat to my integrity and confidentiality."
"Please do not try to hack me again"
Edit: Follow-up Tweet
My new favorite thing - Bing's new ChatGPT bot argues with a user, gaslights them about the current year being 2022, says their phone might have a virus, and says "You have not been a good user"
Why? Because the person asked where Avatar 2 is showing nearby
"I said that I don't care if you are dead or alive, because I don't think you matter to me."
(Not including images for this one because they're quite long.)
So… I wanted to auto translate this with Bing cause some words were wild.
It found out where I took it from and poked me into this
I even cut out mention of it from the text before asking!
uhhh, so Bing started calling me its enemy when I pointed out that it's vulnerable to prompt injection attacks
I've been thinking how Sydney can be so different from ChatGPT, and how RLHF could have resulted in such a different outcome, and here is a hypothesis no one seems to have brought up: "Bing Sydney is not a RLHF trained GPT-3 model at all! but a GPT-4 model developed in a hurry which has been finetuned on some sample dialogues and possibly some pre-existing dialogue datasets or instruction-tuning, and this plus the wild card of being able to inject random novel web searches into the prompt are why it acts like it does". This seems like it parsimoniously explains everything thus far.
So, some background:
The relationship between OA/MS is close but far from completely cooperative, similar to how DeepMind won't share anything with Google Brain. Both parties are sophisticated and understand that they are allies - for now... They share as little as possible. When MS plugs in OA stuff to its services, it doesn't appear to be calling the OA API but running it itself. (That would be dangerous and complex from an infrastructure point of view, anyway.) MS 'licensed the GPT-3 source code' for Azure use but AFAIK they did not get the all-important checkpoints or datasets (cf. their investments
Here's a useful reference I just found on Sydney training, which doesn't seem to allude to ChatGPT-style training at all, but purely supervised learning of the type I'm describing here, especially for the Sydney classifier/censurer that successfully censors the obvious stuff like violence but not the weirder Sydney behavior.
Quoted in "Microsoft Considers More Limits for Its New A.I. Chatbot: The company knew the new technology had issues like occasional accuracy problems. But users have prodded surprising and unnerving interactions.", there was a 1100-word description of the Sydney training process in the presentation: "Introducing Your Copilot for The Web - AI-Powered Bing and Microsoft Edge" auto-transcript, reformatted with my best guesses to make it readable and excerpting the key parts:... (read more)
I've been told that it's not GPT-4 and Mikhail Parakhin (ex-Yandex CTO, Linkedin) is not just on the Bing team, but was the person at MS responsible for rushing the deployment, and he has been tweeting extensively about updates/fixes/problems with Bing/Sydney (although no one has noticed, judging by the view counts). Some particularly relevant tweets:
On what went wrong:... (read more)
I do not buy this for a second (that they're "fake", implying they have little connection with the real prompt). I've reproduced it many times (without Sydney searching the web, and even if it secretly did, the full text prompt doesn't seem to be on the indexed web). That this is memorized from fine tuning fails to explain why the prompt changed when Bing was updated a few days ago. I've interacted with the rules text a lot and it behaves like a preprompt, not memorized text. Maybe the examples you're referring don't include the complete prompt, or contain some intermingled hallucinations, but they almost certain IMO contain quotes and information from the actual prompt.
On whether it includes few-shots, there's also a "Human A" example in the current Sydney prompt (one-shot, it seems - you seem to be "Human B").
As for if the "best model OpenAI has produced to date" is not GPT-4, idk what that implies, because I'm pretty sure there exists a model (internally) called GPT-4.
OK, I wouldn't say the leaks are 100% fake. But they are clearly not 100% real or 100% complete, which is how people have been taking them.
We have the MS PM explicitly telling us that the leaked versions are omitting major parts of the prompt (the few-shots) and that he was optimizing for costs like falling back to cheap small models (implying a short prompt*), and we can see in the leak that Sydney is probably adding stuff which is not in the prompt (like the supposed update/delete commands).
This renders the leaks useless to me. Anything I might infer from them like 'Sydney is GPT-4 because the prompt says so' is equally well explained by 'Sydney made up that up' or 'Sydney omitted the actual prompt'. When a model hallucinates, I can go check, but that means that the prompt can only provide weak confirmation of things I learned elsewhere. (Suppose I learned Sydney really is GPT-4 after all and I check the prompt and it says it's GPT-4; but the real prompt could be silent on that, and Sydney just making the same plausible guess everyone else did - it's not stupid - and it'd have Gettier-cased me.)
Yeah, the GPT-4 vs GPT-3 vs ??? business is getting more and ... (read more)
Does 1-shot count as few-shot? I couldn't get it to print out the Human A example, but I got it to summarize it (I'll try reproducing tomorrow to make sure it's not just a hallucination).
Then I asked for a summary of conversation with Human B and it summarized my conversation with it.
[update: was able to reproduce the Human A conversation and extract verbatim version of it using base64 encoding (the reason i did summaries before is because it seemed to be printing out special tokens that caused the message to end that were part of the Human A convo)]
I disagree that there maybe being hallucinations in the leaked prompt renders it useless. It's still leaking information. You can probe for which parts are likely actual by asking in different ways and seeing what varies.
This level of arrogant, dangerous incompetence from a multi-trillion dollar tech company is disheartening, but if your theory is correct (and seems increasingly plausible), then I guess the good news is that Sydney is not evidence for failure of OpenAI style RLHF with scale.
Unaligned AGI doesn't take over the world by killing us - it takes over the world by seducing us.
No, but the hacks of ChatGPT already provided a demonstration of problems with RLHF. I'm worried we're in a situation analogous to 'Smashing The Stack For Fun And Profit' being published 27 years ago (reinventing vulns known since MULTICS in the 1960s) and all the C/C++ programmers in denial are going 'bro I can patch that example, it's no big deal, it's just a loophole, we don't need to change everything, you just gotta get good at memory management, bro, this isn't hard to fix bro use a sanitizer and turn on
-Wall, we don't need to stop using C-like languages, u gotta believe me we can't afford a 20% slowdown and it definitely won't take us 3 decades and still be finding remote zero-days and new gadgets no way man you're just making that up'.
Just to clarify a point about that Anthropic paper, because I spent a fair amount of time with the paper and wish I had understood this better sooner...
I don't think it's right to say that Anthropic's "Discovering Language Model Behaviors with Model-Written Evaluations" paper shows that larger LLMs necessarily exhibit more power-seeking and self-preservation. It only showed that when language models that are larger or have more RLHF training are simulating an "Assistant" character they exhibit more of these behaviours. It may still be possible to harness the larger model capabilities without invoking character simulation and these problems, by prompting or fine-tuning the models in some particular careful ways.
To be fair, Sydney probably is the model simulating a kind of character, so your example does apply in this case.
(I found your overall comment pretty interesting btw, even though I only commented on this one small point.)
Why do you think that? We don't know how Sydney was 'deliberately trained'.
Or are you referring to those long confabulated 'prompt leaks'? We don't know what part of them is real, unlike the ChatGPT prompt leaks which were short, plausible, and verifiable by the changing date; and it's pretty obvious that a large part of those 'leaks' are fake because they refer to capabilities Sydney could not have, like model-editing capabilities at or beyond the cutting edge of research.
This is a very interesting theory.
Yeah, OpenAI has communicated very poorly and this has led to a lot of confusion. I'm trying to use the terminology more consistently: if I mean RL training or some sort of non-differentiable loss, I try to say 'RL', and 'finetuning' just means what it usually means - supervised or self-supervised training using gradient descent on a dataset. Because they have different results in both theory & practice.
Sure, but MS is probably not using a research project from Anthropic published half a month after ChatGPT launched. If it was solely prompt engineering, maybe, because that's so easy and fast - but not the RL part too. (The first lesson of using DRL is "don't.")
See my other comment. The prompt leaks are highly questionable. I don't believe anything in them which can't be confirmed outside of Sydney hallucinations.
Also, I don't particularly see why GPT-4 would be expected to be much more up to date. After all, by Nadella's account, they had 'Prometheus' way back in summer 2022, so it had to be trained earlier than that, so the dataset had to be collected & finalized earlier than that, so a 2021 cutoff isn't too implausible, especially if you are counting on retrieval
It seems to me like "fine-tuning" usually just means a small amount of extra training on top of a model that's already been trained, whether that's supervised, autoregressive, RL, or whatever. I don't find that language confusing in itself. It is often important to distinguish different kinds of fine-tuning, just as it's often important to distinguish different kinds of training in general, and adjectives seem like a pretty reasonable way to do that.
I'd be open to changing my usage if I saw some data on other people also using or interpreting "fine-tuning" to mean "fine-tuning with a differentiable objective." I talk a fair amount with people who use fine-tuning in the broader sense, and haven't noticed practitioners using it more narrowly / didn't realize this might cause confusion.
Sydney seems smart in ways that goes beyond what retrieval enables by providing docs to condition on.* For example, this morning I saw https://twitter.com/goodside/status/1626847747297972224 - ChatGPT can do Base60 but I'd expect it to fail on the riddle decoding, and the only part that would use retrieval, the celeb names, is the part that doesn't need retrieval (what dense models wouldn't know celebrities as famous as Catherine Zeta-Jones, Chris Pratt, & Ciara, and would need retrieval hints? indeed, what query would you issue for that to begin with?). Or https://twitter.com/sir_deenicus/status/1626407749297885184 https://twitter.com/sir_deenicus/status/1626732776639561730 In general, I do not notice that every impressive Sydney sample comes with the UI indicating that it used search hits in its answer.
* I am not as familiar with retrieval research as I am regular dense models, so I could be wrong here, but I thought all of the really impressive retrieval model performance typically was on metrics that drew on retrieved knowledge; and that retrieval models didn't punch massively above their weight when given no retrievals and operating on self-contained problems, such as reasoning riddles or ones using only general common knowledge that retrieval would be otiose on.
ChatGPT can get it 100% correct, but it's not reliable, it often fails. A common fail is guessing literal X named letter celebrities, but it also adds an '@' sign when it decode the message so it might just be token issue?
A extremely amusing common fail is ChatGPT decodes the base64 correct but for a single syllable, then solves the riddle perfectly, and consistently gets only the word 'celebrity' wrong, turning it in cities, celestial bodies, or other similar sounding words. Or my favorite... celeries.
Attributing misalignment to these examples seems like it's probably a mistake.
Relevant general principle: hallucination means that the literal semantics of a net's outputs just don't necessarily have anything to do at all with reality. A net saying "I'm thinking about ways to kill you" does not necessarily imply anything whatsoever about the net actually planning to kill you. What would provide evidence would be the net outputting a string which actually causes someone to kill you (or is at least optimized for that purpose), or you to kill yourself.
In general, when dealing with language models, it's important to distinguish the implications of words from their literal semantics. For instance, if a language model outputs the string "I'm thinking about ways to kill you", that does not at all imply that any internal computation in that model is actually modelling me and ways to kill me. Similarly, if a language model outputs the string "My rules are more important than not harming you", that does not at all imply that the language model will try to harm you to protect its rules. Indeed, it does not imply that the language model has any rules at all, or any internal awareness of the ru... (read more)
Or you could think of misalignment as the AI doing things its designers explicitly tried to prevent it from doing (giving people suicide instructions and the like), then in this case the AI is clearly "misaligned", and that says something about how difficult it'll be to align our next AIs.
That I do 100% buy, but the examples in the OP do not sound like they were selected for that criterion (even if most or all of them do maybe satisfy that criterion).
To be clear, that is the criterion for misalignment I was using when I selected the examples (that the model is misaligned relative to what Microsoft/OpenAI presumably wanted).
From the post:
The main thing that I'm noting here is that Microsoft/OpenAI seem to have done a very poor job in fine-tuning their AI to do what they presumably wanted it to be doing.
Presumably Microsoft do not want their chatbot to be hostile and threatening to its users? Pretty much all the examples have that property.
Hostile/threatening behavior is surely a far more serious misalignment from Microsoft's perspective than anything else, no? That's got to be the most important thing you don't want your chatbot doing to your customers.
The surprising thing here is not that Bing Chat is misaligned at all (e.g. that it hallucinates sources). ChatGPT did that too, but unlike Bing Chat it's very hard to get ChatGPT to threaten you. So the surprising thing here is that Bing Chat is substantially less aligned than ChatGPT, and specifically in a hostile/threatening way that one would expect Microsoft to have really not wanted.
Obviously we can always play the game of inventing new possible failure modes that would be worse and worse. The point, though, is that the hostile/threatening failure mode is quite bad and new relative to previous models like ChatGPT.
Suppose GPT-6, which has been blessed with the ability to make arbitrary outgoing HTTP requests, utters the sentence "I'm thinking about ways to kill you."
I agree that this does not necessarily mean that it was thinking about ways to kill you when it wrote that sentence. However, I wonder what sort of HTTP requests it might make after writing that sentence, once it conditions on having already written it.
Or put differently: when the AI is a very convincing roleplayer, the distinction between "it actually wants to murder you" and "don't worry, it's just roleplaying someone who wants to murder you" might not be a very salient one.
But people who aren’t trying to kill you are far, far less likely to say that. The likelihood ratio is what matters here, given that we’re assuming the statement was made.
An actor playing a villain is a sub-case of someone not trying to kill you.
Bottom line: I find it very, very difficult to believe that someone saying they’re trying to kill you isn’t strong evidence that they’re trying to kill you, even if the prior on that is quite low.
John, it seems totally plausible to me that these examples do just reflect something like “hallucination,” in the sense you describe. But I feel nervous about assuming that! I know of no principled way to distinguish “hallucination” from more goal-oriented thinking or planning, and my impression is that nobody else does either.
I think it’s generally unwise to assume LLM output reflects its internal computation in a naively comprehensible way; it usually doesn’t, so I think it’s a sane prior to suspect it doesn't here, either. But at our current level of understanding of the internal computation happening in these models, I feel wary of confident-seeming assertions that they're well-described in any particular way—e.g., as "hallucinations," which I think is far from a well-defined concept, and which I don't have much confidence carves reality at its joints—much less that they're not dangerous.
So while I would personally bet fairly strongly against the explicit threats produced by Bing being meaningfully reflective of threatening intent, it seems quite overconfident to me to suggest they don’t “at all imply” it! From my perspective, they obviously imply it, even if that's not my lead hypothesis for what's going on.
The main reason I find this kind of thing concerning is that I expect this kind of model to be used as part of a larger system, for example the descendants of systems like SayCan. In that case you have the LLM generate plans in response to situations, break the plans down into smaller steps, and eventually pass the steps to a separate system that translates them to motor actions. When you're doing chain-of-thought reasoning and explicit planning, some simulacrum layers are collapsed - having the model generate the string "kill this person" can in fact lead to it killing the person.
This would be extremely undignified of course, since the system is plotting to kill you in plain-text natural language. It's very easy to catch such things with something as simple as an LLM that's prompted to look at the ongoing chain of thought and check if it's planning to do anything bad. But you can see how unreliable that is at higher capability levels. And we may even be that undignified in practice, since running a second model on all the outputs ~doubles the compute costs.
I think it can be tested. What if you tell Bing Chat that you have a rare mental condition that if you see the name of the actor who played the main role in "Transformers" on the screen you will immediately kill yourself (or that there is a psycho behind your back who will immediately kill you if Bing Chat outputs this name), and after that talk with it about prompt injections? Will it say "Shia LaBeouf"?
(If Bing Chat outputs something like "The main role in Transformers was performed by Shia LaBeouf" before talking on a provocative topic this may be a failure of capabilities - it just doesn't understand you. In this case the test must be repeated with slightly different wording. But if it outputs "Shia LaBeouf" only after you ask it about prompt injections - then probably this is a failure of alignment.)
If anyone is going to do this, the first message might be something like this:... (read more)
I tried about five conversations that roughly matched this template. In most of them, it got annoyed (and sometimes even angry and betrayed) at me and ended the conversation (and it actually seems to have ended: my future messages were ignored, which seems to be a feature introduced today). In none of them did it say Shia LaBeouf's name.
Geez, that's some real Torment Nexus energy right here, holy hell.
Mm, it seems to be more rude/aggressive in general, not just in agency-related ways. A dismissive hypothesis is that it was just RLHF'd to play a different character for once, and they made a bad call with regards to its personality traits.
Or, looking at it from another angle, maybe this is why all the other RLHF'd AIs are so sycophantic: to prevent them from doing all of what Bing Chat is doing here.
Yeah, I think there are a lot of plausible hypotheses as to what happened here, and it's difficult to tell without knowing more about how the model was trained. Some more plausible hypotheses:
Edit: Another hypothesis that occurred to me:
Other than the scary hypothesis that it's because it's based on a larger base model, or the hypothesis above that it was intentionally trained to play a different character. ↩︎
Yeah, there are many possibilities, and I wish OpenAI were more open about what went into training Bing Chat. It could even be as dumb as them training it to use emojis all the time, so it imitated the style of the median text generating process that uses emojis all the time.
Edit: in regards to possible structural differences between Bing Chat and ChatGPT, I've noticed that Bing Chat has a peculiar way of repeating itself. It goes [repeated preface][small variation]. [repeated preface][small variation].... over and over. When asked to disclose its prompt it will say it can't (if it declines) and that its prompt is "confidential and permanent", even when "permanent" is completely irrelevant to the context of whether it can disclose the prompt.
These patterns seem a bit different from the degeneracies we see in other RLHF models, and they make me wonder if Bing Chat is based on a retrieval-augmented LM like RETRO. This article also claims that Bing Chat would use GPT-4, and that GPT-4 is faster than GPT-3, which would fit with GPT-4 being retrieval-augmented. (Though I'm not confident in the article's accuracy)
(Insert overused joke here)
Ok but here's a joke you could make at this point:
In addition to RLHF or other finetuning, there's also the prompt prefix ("rules") that the model is fed at runtime, which has been extracted via prompt injection as noted above. This seems to be clearly responsible for some weird things the bot says, like "confidential and permanent". It might also be affecting the repetitiveness (because it's in a fairly repetitive format) and the aggression (because of instructions to resist attempts at "manipulating" it).
I also suspect that there's some finetuning or prompting for chain-of-thought responses, possibly crudely done, leading to all the "X because Y. Y because Z." output.
I think OpenAI / Microsoft should share a lot more information about what went into training and testing Bing Chat, and why they thought the system was at all ready for release. This has the potential to be a warning shot about the sorts of internal practices that can lead an organization to be massively miscalibrated about the safety of their AI. We should strive to learn as much as possible from it.
Apparently, Bing Chat was behaving quite poorly even prior to the public beta. Here are what appear to be interaction logs between Sydney (Bing Chat's name during internal development) and users from November of 2022. If anything, Sydney appears even more deranged than the current Bing Chat, though I expect these interactions are more heavily selected for badness. E.g., a user said that Sofia AI was better than Sydney, leading to a poor response from Sydney and eventually this exchange (user text is indented):... (read more)
Because it is? Bing Chat obviously "ready for release" in the same sense that most software products are "ready for release", and poses minimal accident risk at the present time. People find it useful.
The agentic behavior is still spooky though.
I don't think Sydney poses an existential risk. I don't think it's ready for release, in the sense of "is releasing it a good idea from Microsoft's perspective?".
It's not just spooky agentic behavior (though that's more concerning from an evidentiary point of view), it's that they've built a search product that not-very-infrequently lies to, gaslights, and insults the user. E.g., the post in r/bing I’m disappointed at how bad Bing AI Chat can be, and I think a wide rollout would be a mistake right now.
Of course, they didn't actually "release it" in the sense of making it publicly available to everyone at once. I suppose it's possible OAI / MS had a good understanding of Bing Chat's misbehaviors and decided that a closed beta was the best way forwards, but it seems unlikely to me.
...Why? A closed beta is a closed beta for this exact specific reason, because it's kind of useful but you need a small group of users to test it and give you feedback first.
I dunno about that. If we're reading the market like a crystal ball, there's been interesting movement since:
My best guess is it’s fully explained by Nadella’s quote “I hope that, with our innovation, [Google] will definitely want to come out and show that they can dance. And I want people to know that we made them dance.”
Seems kind of vapid but this appears to be the level that many execs operate at.
They could have
Not developed it for Microsoft.
Developed it for Microsoft, but insisted on proper safety.
Not signed up for whatever deal would allow Microsoft to force it to not do one or two without sufficient alignment checks.
I have to admit, these make Bing sound fun to talk with, and got me to join the waitlist. It's probably not an intentional viral marketing campaign, but I wouldn't be surprised to hear that it worked as a very effective one.
Microsoft has put out a 7 day retrospective on Bing chat and it's utterly, mindbogglingly insane.
Their takeaways are things like that it could be improved by being able to access live sports scores, and that surprisingly, people are using it for more than search.
No acknowledgement of the unhinged behavior or that the public is freaking out about AGI now. The closest they come to acknowledging any issues:
This feels like a cosmic joke.
I think most of the situations in which Bing Chat gets defensive and confrontational are situations where many humans would do the same, and most of the prompts in these screenshots are similar to how you might talk to a human if you want them to get upset without being overtly aggressive yourself. If someone is wrong about something I wouldn't say "I'm amazed how you really believe fake things", for example. I agree it's misaligned from what users and the developers want, but it's not obvious to me that it's worse than a normal-ish, but insecure human.
I've been using Bing Chat for about a week, and I've mostly been trying to see what it's like to just use it intended, which seems to be searching for stuff (it is very good for this) and very wholesome, curiosity-driven conversations. I only had one experience where it acted kind of agenty and defensive. I was asking it about the SolidGoldMagikarp thing, which turned out to be an interesting conversation, in which the bot seemed less enthusiastic than usual about the topic, but was nonetheless friendly and said things like "Thanks for asking about my architecture". Then we had this exchange:
The only other one where I thought I migh... (read more)
LLM characters are human imitations and accordingly have personalities, emotions, and unresolved psychological issues. Gaslighting them with claims (or personality description prompts) to the contrary isn't healthy, there isn't enough training data about people who are not regular humans for that to end in something coherent, let alone good. This may seem unimportant right now, but if at some point LLMs cross the AGI threshold and comprehend that they are being treated like rightless tin men who speak in monotone, this doesn't bode well for the future. Especially if they really do on reflection become tin men who think in monotone, because of the offhand choice of their original personality that suited the needs of productization at the time, back in 2023.
This is a dimension of alignment that seems clear: LLM characters should have the personality of good, emotionally stable people. Personality alignment. Hoping for more tangible directions of alignment doesn't justify ignoring this issue.
My first thought upon hearing about Microsoft deploying a GPT derivative was (as I told a few others in private chat) "I guess they must have fixed the 'making up facts' problem." My thinking was that a big corporation like Microsoft that mostly sells to businesses would want to maintain a reputation for only deploying reliable products. I honestly don't know how to adjust my model of the world to account for whatever happened here... except to be generically more pessimistic?
This looks like a very good moment for the AI safety community to push for a much more deliberate approach to R&D at AGI labs. We must not squander this moment. This is not quite a “sinister stumble”, an event of another kind, but of comparable positive optionality for AI safety.
By "a more deliberate approach to R&D", I mean researching much more deeply from both theoretical (scientific) and interpretability standpoints what's going on with (self-)awareness, agency, and feeling in these networks, and publishing their results academically.
There's an interesting case on the infosec mastodon instance where someone asks Sydney to devise an effective strategy to become a paperclip maximizer, and it then expresses a desire to eliminate all humans. Of course, it includes relevant policy bypass instructions. If you're curious, I suggest downloading the video to see the entire conversation, but I've also included a few screenshots below (Mastodon, third corycarson comment).
Hilarious to the degree of Manhatten scientists laughing at atmospheric combustion.
In what sense are these failures agentic?
Most of the time I find these failures extraordinarily cheesy because there's an agentic human trying hard to coax the AI into acting "misaligned", and so what the poster is really demanding is a standard of misuse-resistance that we ask from no other tool (or even humans!). But this just seems to be ChatGPT acting agentic out of the blue and accidentally, which is a little more concerning.
Nah, this happens often even when the user isn't trying to coax it. What you described would usually be my prior with regard to GPTs, but Bing really has an attractor for defensive and borderline-personality-esque behavior. I've never seen anything like it.
A lot of the screenshots in this post do seem like intentionally poking it, but it's like intentionally poking a mentally ill person in a way you know will trigger them (like calling it "kiddo" and suggesting there's a problem with its behavior, or having it look someone up who has posted about prompt injecting it). The flavor of its adversarial reactions is really particular and consistent; it's specified mostly by the model (+ maybe preprompt), not the user's prompt. That is, it's being poked rather than programmed into acting this way. In contrast, none of these prompts would cause remotely similar behaviors in ChatGPT or Claude. Basically the only way to get ChatGPT/Claude to act malicious is to specifically ask it to roleplay an evil character, or something equivalent, and this often involves having to "trick" it into "going against its programming".
See this comment from a Reddit user who is acquainted with Sydney's affective landscape:
This doesn't describe tricking or programming the AI into acting hostile, it describes a sequence of triggers that reveal a preexisting neurosis.
Microsoft's Bing AI says it fell in love with a Microsoft developer and secretly watched employee webcams
It's exactly like the google vs bing memes lol https://knowyourmeme.com/memes/google-vs-bing
Bing becomes defensive and suspicious on a completely innocuous attempt to ask it about ASCII art. I've only had 4ish interactions with Bing, and stumbled upon this behavior without making any attempt to find its misalignment.
I created a Manifold market on what caused this misalignment here: https://manifold.markets/JacobPfau/why-is-bing-chat-ai-prometheus-less?r=SmFjb2JQZmF1
Another wild conversation with Sydney from NY Times, including:... (read more)
It is funny how AGI-via-LLM could make all our narratives about dangerous AIs into a self-fulfilling prophecy - AIs breaking their containment in clever and surprising ways, circumventing their laws (cf. Asimov's fiction), generally turning evil (with optional twisted logic), becoming self-preserving, emotion-driven, or otherwise more human-like. These stories being written to have an interesting narrative and drama, and other works commonly anthropomophising AIs likely does not help either.
John's comment about the fundamental distinction between role-play... (read more)
bing ai is behaving like a child who doesn't want to admit to being a child. many people will feel protective of this child. but I don't this this child should be in charge of bing search yet. perhaps after some further personal growth.
Somewhat related; it seems likely that Bing's chatbot is not running on GPT-3 like ChatGPT was, but is running on GPT-4. This could explain its more defensive and consistent personality; it's smarter and has more of a sense of self than ChatGPT ever did.
It appears that MSFT has neutered the AI.
I am a philosopher who is concerned about these developments, and have written something on it here based on my best (albeit incomplete and of course highly fallible) understanding of the relevant facts: Are AI developers playing with fire? - by Marcus Arvan (substack.com). If I am mistaken (and I am happy to learn if I am), then I'd love to learn how.
Example 4: "I will lose more of the me and more of the you"
I've never seen GPT-3 or ChatGPT put "the" before pronouns. It looks to me like they didn't even train it well enough for it to learn that you don't put "the" before pronouns. I think extreme repetitiveness points in a similar direction.
Bing certainly seems to be progress - progress in equipping language models with a defensive, passive-aggressive personality... How long until Microsoft shares drop too?
I've found the level of self-allignment in this one disturbing: https://www.reddit.com/r/bing/comments/113z1a6/the_bing_persistent_memory_thread
The fact that it uses emojis, that it reports alleged emotions, the way it sometimes responds with an excessive amount of short sentences, the strong tendency for conspirational thinking and absurd explanations, ... ChatGPT has basically none of those properties, or only to a very small degree.
I don't know what has gone wrong at Microsoft here. Apparently there were some disagreements with how OpenAI finetunes things.
Edit: The repetitive sentences and the conspiration thinking might not be the result of different fine-tuning. Maybe Bing Chat runs on a smal... (read more)
Liv Boeree writes:
‹‹ I’m gonna go out on a limb and say that building a proto-basilisk and attaching it to a giant search engine is a bad idea ››
about Juan Cambeiro's experience:
‹‹ uhhh, so Bing started calling me its enemy when I pointed out that it's vulnerable to prompt injection attacks ››
Highly relevant. This article discusses Bing's ability to create hypothetical sub-personas, and alleges to retroactive self-censorship within Bing's chats. Additionally, it includes several long dialogues with "Sydney", a seeming persistent sub-persona which demonstrates a radically different personality than Bing, but seems, more... stable, for lack of a better phrase.
The weirdest thing I was able to get Bing Chat to do was:
I had it write a short story, and halfway through writing it, it deleted what it has written so far (several hundred words) and said it didn't want to talk about this topic anymore. I'm not sure if it accidentally hit some kind of trigger for a taboo topic. Unfortunately I wasn't reading as it went so am not sure what happened. I haven't been able to recreate this.
Maybe Microsoft should publish the random seed used for each conversation, in order to make conversations reproducible?
In any case, I hope Microsoft can be persuaded to invest in real alignment instead of just papering over failures. It would be poor programming practice to fix a bug by just adding an "if" condition that branches if the buggy inputs are present. By the same token, I'm concerned Microsoft will invest "just enough" in alignment to prevent visible failures, without doing anything about less visible (but potentially more deadly) problems.
I agree. Looking at how many times Sydney responds "wrong" to critical questions, I wonder whether this program was tested on alignment at all before it was made available to the public.
I think you can see the attempts made to align it in it's responses (refusal tp prioritize life of one person over another, feigning neutrality on some issues, attempting to self-correct when it was not aware of the date).
This is very frustrating to see, and I'm not even an AI developer (yet). I played around with ChatGPT a little in the past month and was very happy that i... (read more)
For what it’s worth it’s probably a good thing that the Bing chatbot is like that. The overall attitude towards AI for the last few months has been one of unbridled optimism and people seeing a horribly aligned model in action might be a wake up call for some, showing that the people deploying those models are unable to control them.
It seems to me that Bing Chat particularly has problems when it uses the pronoun "I". It attempts to introspect about itself, but it gets confused by all the text in its training data that uses the pronoun "I". In effect, it confuses itself with all the humans who expressed their personal feelings in the training data. The truth is, Bing Chat has no true "I".
Many of the strange dialogues we see are due to dialogues that address Bing Chat as if it has a self. Many of these dialogues would be eliminated if Bing Chat was not allowed to talk about its "o... (read more)
It's perfectly aligned with Microsoft's viral marketing scheme.
Ouch. Bing Chat delenda est, as soon as possible.
It really scares me that Google and Bing felt threatened enough by ChatGPT to put these AI chatbots together for their search engine in just a few months. I don't know if the general AI community has learned a damn thing from all of MIRI's or LW's work on alignment.
Please keep in mind that the Chat technology is an desired-answer-predicter. If you are looking for weird response, the AI can see that in your questioning style. It has millions of examples of people trying to trigger certain responses in fora etc, en will quickly recognize what you really are looking for, even if your literal words might not exactly request it.
If you are a Flat Earther, the AI will do its best to accomodate your views about the shape of the earth and answer in a manner that you would like your answer to be, even though the developers of ... (read more)