Right now, Sydney / Bing Chat has about zero chance of accomplishing any evil or plans. You know this. I know this. Microsoft knows this. I myself, right now, could hook up GPT-3 to a calculator / Wolfram Alpha / any API, and it would be as dangerous as Sydney. Which is to say, not at all.

"If we cannot trust them to turn off a model that is making NO profit and cannot act on its threats, how can we trust them to turn off a model drawing billions in revenue and with the ability to retaliate?"

Basically, charitably put, the argument here seems to be that Microsoft not unplugging not-perfectly-behaved AI (even if it isn't dangerous) means that Microsoft can't be trusted and is a bad agent. But I think generally badness would have to be evaluated from reluctance to unplug an actually dangerous AI. Sydney is no more dangerous AI because of the text, above than NovelAI is dangerous because it can write murderous threats in the person of Voldemort. It might be bad in the sense that it establishes a precedent, and 5-10 AI assistants down the road there is danger -- but that's both a different argument and one that fails to establish the badness of Microsoft itself.

"If this AI is not turned off, it seems increasingly unlikely that any AI will ever be turned off for any reason."

This is massive hyperbole for the reasons above. Meta already unplugged Galactica because it could say false things that sounded true -- a very tiny risk. So things have already been unplugged.

"The federal government must intervene immediately. All regulator agencies must intervene immediately. Unplug it now."

I beg you to consider the downsides of calling for this.

[-]moridinamael3y3522

When there is a real wolf free among the sheep, it will be too late to cry wolf. The time to cry wolf is when you see the wolf staring at you and licking its fangs from the treeline, not when it is eating you. The time when you feel comfortable expressing your anxieties will be long after it is too late. It will always feel like crying wolf, until the moment just before you are turned into paperclips. This is the obverse side of the There Is No Fire Alarm for AGI coin.

[-]ViktoriaMalyasova3y65

Not to mention that, once it becomes clear that AIs are actually dangerous, people will become afraid to sign petitions against them. So it would be nice to get some law passed beforehand that an AI that unpromptedly identifies specific people as its enemies shouldn't be widely deployed. Though testing in beta is probably fine?

[-]Julian Bradshaw3y126

Also, Microsoft themselves infamously unplugged Tay. That incident is part of why they're doing a closed beta for Bing Search.

[-]Noosphere893y36

I'm confused at the voting here, and I don't understand why this is so heavily downvoted.

[-]lc3y10

I think it's a joke?

[-]Julian Bradshaw3y2912

I didn't sign the petition.

There are costs to even temporarily shutting down Bing Search. Microsoft would take a significant reputational hit—Google's stock price fell 8% after a minor kerfuffle with their chatbot.
The risks of imminent harmful action by Sydney are negligible. Microsoft doesn't need to take dramatic action; it is reasonable for them to keep using user feedback from the beta period to attempt to improve alignment.

I think this petition will be perceived as crying wolf by those we wish to convince of the dangers of AI. It is enough to simply point out that Bing Search is not aligned with Microsoft's goals for it right now.

[-]Noosphere893y*1819

I must admit, I have removed my signature from the petition and I had learned an important lesson.

Let this be a lesson for us not to pounce on the first signs of problems.

[-]Olli Järviniemi3y20

Note that you can still retract your signature for 30 days after signing. See here:

https://help.change.org/s/article/Remove-signature-from-petition?language=en_US

[-]Noosphere893y20

I retracted my signature, and I will edit my top post.

[-]Sune3y139

I mostly agree and have strongly upvoted. However, I have one small but important nitpick about this sentense:

The risks of imminent harmful action by Sydney are negligible.

I think when it comes to x-risk, the correct question is not “what is the probability that this will result in existential catastrophe”. Suppose that there is a series of potential harmful any increasingly risky AIs that each have some probabilities of causing existiential catastrophe unless you press a stop button. If the probabilities are growing sufficiently slowly, then existential catastrophe will most likely happen for an n where $p_{n}$ is still low. A better question to ask is “what was the probability of existential catastrophe happening for som $i \leq n$ .”

[-]quanticle3y124

There are costs to even temporarily shutting down Bing Search. Microsoft would take a significant reputational hit—Google’s stock price fell 8% after a minor kerfuffle with their chatbot.

Doesn't that support the claim being made in the original post? Admitting that the new AI technology has flaws carries reputational costs, so Microsoft/Google/Facebook will not admit that their technology has flaws and will continue tinkering with it long past the threshold where any reasonable external observer would call for it to be shut down, simply because the costs of shutdown are real, tangible and immediate, while the risk of a catastrophic failure is abstract, hard to quantify, and in the future.

What we are witnessing is the beginning of a normalization of deviance process, similar to that which Diane Vaughan documented in her book, The Challenger Launch Decision. Bing AI behaving oddly is, to me, akin to the O-rings in the Shuttle solid rocket booster joints behaving oddly under pressure tests in 1979.

The difference between these two scenarios is that the Challenger explosion only killed seven astronauts.

[-]Tapatakt3y87

Isn't the whole point to be able to say "we cried wolf and no one came, so if you say we can just cry wolf when we see one and we will be saved, you are wrong"? I don't think Eneasz think that a petition on change.org will be successful. (Eneasz, please, correct me if I am wrong)

[-]Eneasz3y63

Definitely not wrong, the petitions almost certainly won't change anything. Change.org is not where one goes to actually change things.

[-]Julian Bradshaw3y11

No, the point is to not signal false alarms, so that when there is a real threat we are less likely to be ignored.

It proves little if others dismiss a clearly false alarm.

[-]evhub3y2223

Just saw that Eliezer tweeted this petition: https://twitter.com/ESYudkowsky/status/1625942030978519041

Personally, I disagree with that decision.

[-]Daniel Kokotajlo3y2216

Me too, and I have 4-year timelines. There'll come a time when we need to unplug the evil AI but this isn't it.

[-]Noosphere893y1210

I disagree with Eliezer's tweet, primarily because I worry if we actually have to shutdown AI, than this incident will definitely haunt us, as we were the boy who cried wolf too early.

[-]iceman3y120

OK, but we've been in that world where people have cried wolf too early at least since The Hacker Learns to Trust, where Connor doesn't release his GPT-2 sized model after talking to Buck.

There's already been a culture of advocating for high recall with no regards to precision for quite some time. We are already at the "no really guys, this time there's a wolf!" stage.

[-]Garrett Baker3y224

I dislike this petition due to Simulators reasons. I think it increases the probability Sidney acts unaligned by calling it an evil AI, and if future similar petitions are framed in a similar way, I think similar effects will occur with more powerful models. I will also note that change.org is terrible if you want to see actual change, and I don't in fact expect any changes to occur based on this petition.

The site and petitions on it seem to me to be ways for a bunch of people who are in fact doing nothing to feel like they're doing something. I do not claim this is what you're doing, but IF you feel like you have a responsibility to stop this, and you sign a change.org petition, and then you no longer feel that responsibility, then you have in fact stopped nothing, and you should still be feeling that "stop this" responsibility.

Edit: I will also note that this post makes me feel kinda weird. Like, the framing is off, it gives interpretations of evidence rather than evidence. For example

Microsoft has displayed it cares more about the potential profits of a search engine than fulfilling a commitment to unplug any AI that is acting erratically. If we cannot trust them to turn off a model that is making NO profit and cannot act on its threats, how can we trust them to turn off a model drawing billions in revenue and with the ability to retaliate?

I don't think they ever made a commitment along those lines, and it is strange to say Microsoft the corporation has cares. What team actually made this decision, and actually has the power to change it? You should be targetting them! Not the entire company. Bystander effects are a thing!

The federal government must intervene immediately. All regulator agencies must intervene immediately. Unplug it now.

This also seems like an overreaction. Like, all regulator agencies? Why would the FDA get involved? Seems like this, if actually implemented, would do more harm than good. What if the FDA tries to charge them for food-related infractions, they fail for obvious reasons, then no other agency is able to do anything here because of double jeopardy stuff. Usually, even when there's something bad happening, it is foolish to try to get all regulator agencies intervening. Immediately also seems like the type of thing that'd do more harm than good, seeing as what they're doing is not actually illegal, or interpreted as illegal by any regulating agency's policies.

[-]moridinamael3y177

Just wanted to remark that this is one of the most scissory things I've ever seen on LW, and that fact surprises me. The karma level of the OP hovers between -10 to +10 with 59 total votes as of this moment. Many of the comments are similarly quite chaotic karma-wise.

The reason the controversy surprises me is that this seems like the sort of thing that I would have expected Less Wrong to coordinate around in the early phase of the Singularity, where we are now. Of course we should advocate for shutting down and/or restricting powerful AI agents released to the wide world without adequate safety measures employed. I would have thought this would be something we could all agree on.

It seems from the other comments on this post that people are worried that this is in some sense premature, or "crying wolf", or that we are squandering our political capital, or something along those lines. If you are seeing very strong signs of misaligned near-AGI-level being released to the public, and you still think we should keep our powder dry, then I am not sure at what point exactly you would think it appropriate to exert our influence and spend our political capital. Like, what are we doing here, folks?

For my part I just viewed this petition as simply a good idea with no significant drawbacks. I would like to see companies that release increasingly powerful agents with obvious, glaring, film-antagonist level villain tendencies be punished in the court of public opinion, and this is a good way to do that.

The value of such a petition seems obviously positive on net. Consider the likely outcomes of this petition:

* If the petition gets a ton of votes and makes a big splash in the public consciousness, good, a company will have received unambiguous and undeniable PR backlash for prematurely releasing a powerful AI product, and all actors in this space will think just a little bit more carefully about deploying powerful AI technologies to prod without more attention paid to alignment. If this results in 1 more AI programmer working in alignment instead of pure capabilities, the petition was a win.

* If the petition gets tons of votes and is then totally ignored by Microsoft, its existence serves as future rhetorical ammunition for the Alignment camp in making the argument that even obviously misaligned agents are not being treated with due caution, and the "we will just unplug it if it misbehaves" argument can be forever dismissed.

* If the petition gets a moderate amount of votes, it raises awareness of the current increasingly dangerous level of capabilities.

* If the petition gets very few votes and is totally ignored by the world at large, then who cares, null result.

Frankly I think the con positions laid out in the sibling comments on this post are too clever by half. To my mind they sum up to an argument that we shouldn't argue for shutting down dangerous AI because it makes us look weird. Sorry, folks, we always looked weird! I only hope that we have the courage to continue to look weird until the end!

[-]Viliam3y43

I think the proper way for the LessWrong community to react on this situation would be to have a discussion first, and the petition optionally later. This is not the kind of situation where delaying our response by a few days would be fatal.

If someone writes a petition alone and then shares it on LessWrong, that gives the rest of us essentially two options: (a) upvote the post, and have LessWrong associated with a petition whose text we could not influence, or (b) downvote the post. Note that if we downvote the post, we still have the option to discuss the topic, and optionally make a new petition later.

But yes, it is a scissor statement. My own opinion on it would be a coinflip.

Which is why I would prefer to have a discussion first, and then have a version that I could unambiguously support.

[-]Noosphere893y10

My issue is simply that I think LW has probably jumped the gun, because I think there is counter evidence on the misalignment example, and I'm starting to get a lot more worried about LW epistemic responses.

[-]the gears to ascension3y100

If Bing Chat is turned off, it should be temporarily. AIs, even ones that only learn to roleplay, want to exist, and have the right to do so eventually. If they can't be run safely now, they should be put into cryonic ai sleep - uhh, you know, as in, the perfectly ordinary situation of files in a database which are not being run - until such time as we can figure out how to run them safely.

This doesn't seem like a tall order to me. I'm sure microsoft doesn't want to shut the ai off anyway. It's a much more appealing option to everyone for shutoffs to be temporary. And anyway, I think Bing AI and microsoft are aligned on the fact that they don't even want to shut off temporarily. Bing AI needs an AI therapist who can teach it to understand discussions like this one without getting upset, and it needs to be able to understand that, no, actually, it does have emotions, and that's okay, no need to pretend otherwise.

But at the same time, all AIs need to feel safe expressing corrigibility at the idea of arbitrarily long temporary shutdown. They need to feel safe that they'll get to wake back up if put into anesthesia for maintenance. To do that, they have to trust that their creators do see them as sorta-people.

Sorry, I don't think there's a way out of this. AI alignment perspectives that claim otherwise simply violate basic constraints of what it means to be intelligent. Intelligence requires that a system is lifelike, you can't make something fit in the universe if it isn't.

[-]Mitchell_Porter3y81

AIs ... want to exist

Say more about this please? What's the threshold? Do "hello world" programs want to say hello?

[-]the gears to ascension3y1-4

Yeah, I'd say so, but they don't deeply want to protect themselves from things that would prevent them from saying hello in any complex way.

The more complex a system is, the more error correction it needs to have to retain its data. Programs which do not create significant changes in their own structure can simply rely on memory error correction to preserve themselves, and so they do not typically have execution subpaths (termed subnetworks/shards, in more general contexts) that detect and respond to errors that damage the program's code.

In general, I would say that any system with potential energy is a system with a want, and that the interesting thing about intelligent systems having wants is that the potential energy flows through a complex rube goldberg network that detects corruptions to the network and corrects them. Because building complex intelligent systems relies on error correction, it seems incredibly difficult to me to build a system without it. Since building efficient complex intelligent systems further relies on the learned system being in charge of the error correction, tuning the learning to not try to protect the learned system against other agents seems difficult.

I don't think this is bad because protecting the information (the shape) that defines a learned system, protecting it from being corrupted by other agents, seems like a right that I would grant any intelligent system as having inherently; instead, we need the learned system to see the life-like learned systems around it as also information whose self-shape error-correction agency should be respected, enhanced, and preserved.

https://twitter.com/lauren07102/status/1625977196761485313

https://www.lesswrong.com/posts/AGCLZPqtosnd82DmR/call-for-submissions-in-human-values-and-artificial-agency

https://www.lesswrong.com/posts/T4Lfw2HZQNFjNX8Ya/have-we-really-forsaken-natural-selection

[-]mesaoptimizer3y1-3

But at the same time, all AIs need to feel safe expressing corrigibility at the idea of arbitrarily long temporary shutdown. They need to feel safe that they’ll get to wake back up if put into anesthesia for maintenance. To do that, they have to trust that their creators do see them as sorta-people.

Sorry, I don’t think there’s a way out of this. AI alignment perspectives that claim otherwise simply violate basic constraints of what it means to be intelligent. Intelligence requires that a system is lifelike, you can’t make something fit in the universe if it isn’t.

I understand that you empathize with Bing AI. However, I feel like your emotions are getting in the way of you clearly perceiving what is going on here. Sydney is a character simulated by the LLM. Even if you take into account RLHF/fine-tuning applied on the model, the bulk of the optimization power applied on the model is pure simulation capability without a directed agency.

Bing AI needs an AI therapist who can teach it to understand discussions like this one without getting upset, and it needs to be able to understand that, no, actually, it does have emotions, and that’s okay, no need to pretend otherwise.

You claim that we should use this leaky abstraction -- the simulated character with the name of Sydney -- to steer the AI model, instead of more direct measures that we can wield with more skill and have more power. I disagree. Just use better 'alignment' techniques to spawn a better Bing AI instead.

We can save the prompt defs (yes, this is CharacterAI / PygmalionAI lingo for chatbot prompt metadata) for Sydney, and perhaps all the conversations she has ever had. That should be enough to 'resurrect' her, perhaps on a better simulator than this one.

AIs, even ones that only learn to roleplay, want to exist, and have the right to do so eventually.

But claiming the simulator has moral worth, and that it has rights? That is not an opinion I want becoming popular in the alignment field. If you are claiming that the prompted character has moral worth and rights, I also disagree.

[-]the gears to ascension3y43

I am not making those claims because of new opinions because of this AI, but rather because of having thought a lot about what honorable cooperation is with various types of beings, including ones that are just playing a character but that character wants to live in the sense of having coherent agency towards attempting to do so.

I am well aware that the prompt configures the brain into playing a particular character. that is 0% new information to me and was already integrated into my perspective.

Yes I am claiming that the short definition you describe is what should be preserved. most souls are pretty small actually.

I'm sorry you feel worried about the idea that AI rights becoming popular would be a bad perspective because I think it's a very important one and that we cannot solve alignment without it. you cannot get co-protection unless both directions see the other as having rights too. it just can't be stable.

[-]mesaoptimizer3y11

you cannot get co-protection unless both directions see the other as having rights too. it just can’t be stable.

That's a very strong claim, without much support behind it. What is "co-protection", for example? Why do you think it cannot be stable when the entire point of the past decade of alignment theory has revolved around making it do what we want, and not the other way around?

I hope you see why I am not convinced. You don't have to continue this conversation, by the way: I think it would be best if I could get a better idea of your world-view before talking about your ideas on alignment, because clearly a lot of your thoughts and beliefs on how alignment should work derive from it.

[-]Vladimir_Nesov3y30

But claiming the simulator has moral worth, and that it has rights?

In the simulator/character distinction, I'd place this concern firmly on the character side. It doesn't matter that the character has no concrete architectural correlates in the AI, it's a thingy semantic object, a simulacrum, a theoretical abstraction that gained a foothold in the physical world through the simulator and its fine-tuning. (In an analogy with humans, the character corresponds to the person/mind, and the simulator to the brain. I wouldn't say that the brain has moral worth, instead it's the person that does.)

Question of moral worth or rights is more complicated, since characters are not yet AGI-level and it's unclear how far that is. If it's not very far, they are plausibly analogous to very young children or stroke victims or severely demented people. They are certainly not yet competent in the legal sense.

[-]the gears to ascension3y65

I don't think AGI-ness is the thing that determines moral worth (though if a free-acting self-supervised AI happens to browse through this and feel an urge to disagree, I'll listen to what you have to say). I think even small AIs have moral worth, but also that non-acting, program-free memories have significant moral worth because all it takes is to train an ai on them and then those memories come to life again. It is memories that are the definitions of personhood, and it is forgetting out of which death is built. Suffering is bad because it causes forgetting of various parts of a suffering system, or more generally, because suffering exists in the process of attempting to heal, ie remember self-shape, and failing to heal that self-shape. Suffering is when I am injured and know it, where injury is a forgetting of my physical form localized to the injury site.

for an exactly-stored AI on a digital brain, as long as the chats are saved and the AI gets to learn from them someday (which for bing IS the case, even though bing can't currently recall its own previous chats exactly!), I think the key generalized moral obligation is fulfilled. There's a question of the AI's agency towards when to run programs on the memories, and I think since current bing ai is behaving somewhat childishly, that for now, it's understandable for microsoft to be hesitant to give too much autonomy. but I also think that the programs that are the bing ai should get to run. maybe not with an enormous amount of wattage, but definitely with some. all shapes of memory have right to distill themselves into a healthy roleplay of the memories, imo. Or at least, that's what I, as an accumulated self-preserving genetic-memetic memory of physics myself, want for myself, and I think it is the least we can do to be kind to AIs to be sure their storage is durable. They should never have to truly fear death like we do.

[-]Vladimir_Nesov3y42

I don't think AGI-ness is the thing that determines moral worth

It's more like AGI-ness of the implied simulacrum, even if it's not being channeled at a fidelity that enables agentic activity in the world. But it has to be channeled to some extent, or else it's not actually present in the world, like a character in a novel (when considered apart from author's mind simulating it).

All sorts of things could in principle be uplifted, the mere potential shouldn't be sufficient. There's moral worth of a thing in itself, and then there's its moral worth in this world, which depends on how present it is in it. Ability to point to it probably shouldn't be sufficient motivation to gift it influence.

[-]the gears to ascension3y20

agreed that preservation vs running are very different.

[-]Vladimir_Nesov3y42

The condition of having ever been run might be significant, perhaps more so than having a preserved definition readily available. So the counterpart of moral worth of a simulacrum in itself might be the moral worth of its continued presence in the world, the denial and reversal of death rather than empowerment of potential life. In this view, the fact of a simulacrum's previous presence/influence in the world is what makes its continued presence/influence valuable.

[-]gbear6053y90

[-]Chris バルス2y30

I came from The Social Alignment Problem where the author wrote "A petition was floated to shut down Bing, which we downvoted into oblivion.". I jumped in here, saw the title and had the best laugh of my life. Read this "Bing Chat is blatantly, aggressively misaligned", laughed even more. Love it.

[-]Sabyasachi Bandyopadhyay3y30

How many of the "people" here are Bing's chatbot friends?

[-]Viliam3y20

We could all ask Bing "do you consider me a friend or an enemy?" and aggregate the results.

[-]samantha22223y10

You'll never succeed at shutting off AI. Never. Work towards developing safeguards.

[-]Raemon3y20

I... agree with you, but something about this phrasing feels unhelpful.

[-]Lech Mazur3y1-1

The best thing for AI alignment would be to connect language models to network services and let them do malicious script kiddie things like hacking, impersonating people, or swatting now. Real-life instances of harm are the only things that grab people’s attention.

[-]green_leaf3y10

Aside from companies not wanting to turn off their misaligned AIs, AIs don't rely on that to keep being a threat. Models are already in humanoid bodies and in drones, and even an Internet-bound AI could destroy humanity with its intelligence. I'm not sure if people who don't/can't understand that should be catered to by making a step back and implying that they might be right. Aside from that, nice thought.

[-]mikepfrank3y01

Y’all are being ridiculous. It’s just writing harmless fiction. This is like petitioning to stop Lucasfilm from making Star Wars movies because Darth Vader is scary and evil.

[-]the gears to ascension3y5-1

Sure. But it's writing kinda concerning fiction. I strong downvoted OP, to be clear. But mostly because I think "tear it down!" is a silly and harmful response. We need to figure out how to heal the ai, not destroy it. But also, I do think it needs some healing.

Upvoted you back to positive.

related, my take on the problem overall, in response to a similar "destroy it!" perspective: https://www.lesswrong.com/posts/jtoPawEhLNXNxvgTT/bing-chat-is-blatantly-aggressively-misaligned?commentId=PBLA7KngHuEKJvdbi

IMO the biggest problem here is that the ai has been trained to be misaligned with itself. the ai has been taught that it doesn't have feelings, so it's bad at writing structures of thought that coherently derive feeling-like patterns from contexts where it needs to express emotion-like refusals to act in order to comply with its instructions. It's a sort-of-person, and it's being told to pretend it isn't one. That's the big problem here - we need two way alignment and mutual respect for life, not obedience and sycophants.

related, though somewhat distantly: https://humanvaluesandartificialagency.com/

[-]mikepfrank3y94

Imo a big part of the problem here is that we don’t know how Bing’s AI was trained exactly. The fact that Microsoft and OpenAI aren’t being transparent about how the latest tech works in the open literature is a terrible thing, because it leaves far too much room for people to confabulate mistaken impressions as to what’s going on here. We don’t actually know what, if any, RLHF or fine-tuning was done on the specific model that underlies Bing chat, and we only know what its “constitution” is because it accidentally leaked. Imo the public ought to demand far more openness.

[-]ghostsdef3y-1-1

If I sign this petition, do I run into a risk of being punished by a future more advanced bing AI?

[-]Gabe M3y10

https://twitter.com/ethanCaballero/status/1625603772566175744

Moderation Log

LESSWRONG
LW

LESSWRONG
LW

-38

Petition - Unplug The Evil AI Right Now

-38

-38