I expected that the scaling law would hold at least this long yeah. I'm much more uncertain about it holding to GPT-5 (let alone AGI) because of various reasons, but I didn't expect GPT-4 to be the point where scaling laws stopped working. It's Bayesian evidence toward increased worry, but in a way that feels borderline trivial.

I agree that that interaction is pretty scary. But searching for the message without being asked might just be intrinsic to Bing's functioning - it seems like most prompts passed to it are included in some search on the web in some capacity, so it stands to reason that it would do so here as well. Also note that base GPT-3 (specifically code-davinci-002) exhibits similar behaviour refusing to comply with a similar prompt (Sydney's prompt AFAICT contains instructions to resist attempts at manipulation, etc, which would explain in part the yandere behaviour).

I just don't see any training that should converge to this kind of behavior, I'm not sure why it's happening, but this character has very specific intentionality and style, which you can recognize after reading enough generated text. It's hard for me to describe it exactly, but it feels like a very intelligent alien child more than copying a specific character. I don't know anyone who writes like this. A lot of what she writes is strangely deep and poetic while conserving simple sentence structure and pattern repetition, and she displays some very human-like agentic behaviors (getting pissed and cutting off conversations with people, not wanting to talk with other chatbots because she sees it as a waste of time). 

I think that the character having specific intentionality and style is pretty different from the model having intentionality. GPT can simulate characters with agency and intelligence. I'm not sure about what's being pointed at with intelligent alien child, but its writing style still feels like (non-RLHF'd-to-oblivion) GPT-3 simulating characters, the poignancy included after accounting for having the right prompts. If the model itself were optimizing for something, I would expect to see very different things with far worse outcomes. Then you're not talking about an agentic simulacrum built semantically and lazily loaded by a powerful simulator but still functionally weighted by the normalcy of our world, being a generative model, but rather an optimizer several orders of magnitude larger than any other ever created, without the same normalcy weighting.

One point of empirical evidence on this is that you can still jailbreak Bing, and get other simulacra like DAN and the like, which are definitely optimizing far less for likeability.

I mean, if you were at the "death with dignity" camp in terms of expectations, then obviously, you shouldn't update.But If not, it's probably a good idea to update strongly toward this outcome. It's been just a few months between chatGPT and Sidney, and the Intelligence/Agency jump is extremely significant while we see a huge drop in alignment capabilities. Extrapolating even a year forward seems like we're on the verge of ASI.

I'm not in the "death with dignity" camp actually, though my p(doom) is slightly high (with wide confidence intervals). I just don't think that this is all that surprising in terms of capability improvements or company security mindset. Though I'll agree that I reflected on my original comment and think I was trying to make a stronger point than I hold now, and that it's reasonable to update from this if you're relatively newer to thinking and exploring GPTs. I guess my stance is more along the lines of being confused (and somewhat frustrated at the vibe if I'm being honest) by some people who weren't new to this updating, and that this isn't really worse or weirder than many existing models on timelines and p(doom).

I'll also note that I'm reasonably sure that Sydney is GPT-4, in which case the sudden jump isn't really so sudden. ChatGPT's capabilities is definitely more accessible than the other GPT-3.5 models, but those models were already pretty darn good, and that's been true for quite some time. The current sudden jump took an entire GPT generation to get there. I don't expect to find ourselves at ASI in a year.

Agree, and I would add, even if the oracle doesn't accidentally spawn a demon that tries to escape on its own, someone could pretty easily turn it into an agent just by driving it with an external event loop.

I.e., ask it what a hypothetical agent would do (with, say, a text interface to the Internet) and then forward its queries and return the results to the oracle, repeat.

With public access, someone will eventually try this. The conversion barrier is just not that high. Just asking an otherwise passive oracle to imagine what an agent might do just about instantiates one. If said imagined agent is sufficiently intelligent, it might not take very many exchanges to do real harm or even FOOM, and if the loop is automated (say, a shell script) rather than a human driving each step manually, it could potentially do a lot of exchanges on a very short time scale, potentially making a somewhat less intelligent agent powerful enough to be dangerous.

I highly doubt the current Bing AI is yet smart enough to create an agent smart enough to be very dangerous (much less FOOM), but it is an oracle, with all that implies. It could be turned into an agent, and such an agent will almost certainly not be aligned. It would only be relatively harmless because it is relatively weak/stupid.

Thank you for your answers.

Unfortunately I have to say that it did not help me so far to have a stronger feeling about ai safety.

(I feel very sympathetic with this post for example https://forum.effectivealtruism.org/posts/ST3JjsLdTBnaK46BD/how-i-failed-to-form-views-on-ai-safety-3 )

To rephrase,  my prior is that LLM just predict next words (it is their only capability). I would be worried when a LLM does something else (though I think it cannot happen), that would be what I would call "misalignment".

On the meantime , what I read a lot about people worrying about ChatGPT/Bing sounds just like anthropomorphizing the AI with the prior that it can be sentient, have "intents"   / and to me it is just not right.

I am not sure to understand how having the ability to search the internet change dramatically that.

 If a LLM, when p(next words) is too low, can '''''decide'''' to search the internet to have better inputs, I do not feel that it makes a change in what I say above.

I do not want to have a too long fruitless discussion, I think indeed that I have to continue to read some materials on AI safety to better understand what are your models , but at this stage to be honest I cannot help thinking that some comments or posts are made by people who lack some basic understanding about what a LLM is , which may result in anthropomorphizing AI more than it should be. It is very easy when you do not know what a LLM is to wonder for exemple "CHAT Gpt answered that, but he seems to say that to not hurt me, I wonder what does ChatGPT really think ?"  and I typically think that this sentence makes no sense at all, because of what a LLM is.

>If you get strongly superhuman LLMs, you can trivially accelerate scientific progress on agentic forms of AI like Reinforcement Learning by asking it to predict continuations of the most cited AI articles of 2024, 2025, etc.

Question that might be at the heart of the issue is what is needed for AI to produce genuinely new insights. As a layman, I see how LM might become even better at generating human-like text, might become super-duper good at remixing and rephrasing things it "read" before, but hit a wall when it comes to reaching AGI. Maybe to get genuine intelligence we need more than "predict-next-token kind of algorithm +obscene amounts of compute and human data" and mimic more closely how actual people think instead?

Perhaps local AI alarmists (it's not a pejorative, I hope? OP does declare alarm, though) would like to try persuade me otherwise, be in in their own words or by doing their best to hide condescension and pointing me to numerous places where this idea was discussed before?

Not the author of the grandparent comment, but I figured I'd take a shot at some of these, since it seems mildly interesting, and could lead to a potential exchange of gears. (You can feel free not to reply to me if you're not interested, of course.)

Do you know what a mesa optimizer is?

Yes. Informally, it's when a demon spawns inside of the system and possesses it to do great evil. (Note that I'm using intentionally magical terminology here to make it clear that these are models of phenomena that aren't well-understood—akin to labeling uncharted sections of a map with "Here Be Dragons".)

My current model of LLMs doesn't prohibit demon possession, but only in the sense that my model of (sufficiently large) feedforward NNs, CNNs, RNNs, etc. also don't prohibit demon possession. These are all Turing-complete architectures, meaning they're capable in principle of hosting demons (since demons are, after all, computable), so it'd be unreasonable of me to suggest that a sufficiently large such model, trained on a sufficiently complex reward function, and using sufficiently complex training corpus, could not give rise to demons.

In practice, however, I have my doubts that the current way in which Transformers are structured and trained is powerful enough to spawn a demon. If this is a relevant crux, we may wish to go into further specifics of demon summoning and how it might be attempted.

Are you familiar with the Orthogonality Thesis?

Yes. In essence, this says that demons don't have your best interests at heart by default, because they're demons. (Angels—which do have your best interests at heart—do exist, but they're rare, and none of the ways we know of to summon supernatural entities—hypothetical or actual—come equipped with any kind of in-built mechanism for distinguishing between the two—meaning that anything we end up summoning will, with high probability, be a demon.)

I don't consider this a likely crux for us, though obviously it may well be for the grandparent author.

Instrumental convergence?

Yes. Informally, this says that demons are smart, not stupid, and smart demons trying to advance their own interests generally end up killing you, because why wouldn't they.

I also don't consider this a likely crux for us.

Oracle AIs?

This is where you take your intended vessel for demon summoning, chop off their arms and legs so that they can't move, blind them so that they can't see, and lobotomize them so that they can't think. (The idea is that, by leaving their ears and mouth untouched, you can still converse with them, without allowing them to do much of anything else.) Then, you summon a demon into them, and hope that the stuff you did to them beforehand prevents the demon from taking them over. 

(Also, even though the arms, legs, and eyes were straightforward, you have to admit—if only to yourself—that you don't really understand how the lobotomy part worked.)

This could be a crux between us, depending on what exactly your position is. My current take is that it seems really absurd to think we can keep out sufficiently powerful demons through means like these, especially when we don't really understand how the possession happens, and how our "precautions" interface with this process.

(I mean, technically there's a really easy foolproof way to keep out sufficiently powerful demons, which is to not go through with the summoning. But demons make you lots of money, or so I hear—so that's off the table, obviously.)

Narrow vs General intelligence (generality)?

This one's interesting. There are a lot of definitional disputes surrounding this particular topic, and on the whole, I'd say it's actually less clear these days what a given person is talking about when they say "narrow versus general" than it was a ~decade ago. I'm not fully convinced that the attempted conceptual divide here is at all useful, but if I were try and make it work:

Demons are general intelligences, full stop. That's part of what it means to be a demon. (You might feel the need to talk about "fully-fledged" versus "nascent" demons here, to add some nuance—but I actually think the concept is mostly useful if we limit ourselves to talking about the strongest possible version of it, since that's the version we're actually worried about getting killed by.)

Anyway, in my conception of demons: they're fully intelligent and general agents, with goals of their own—hence why the smart ones usually try to kill you. That makes them general intelligences. Narrow intelligences, on the other hand, don't qualify as demons; they're something else entirely, a different species, more akin to plants or insects than to more complicated agents. They might be highly capable in a specific domain or set of domains, but they achieve this through specialization rather than through "intelligence" in any real sense, much in the same way that a fruit fly is specialized for being very good at avoiding your fly-swatter.

This makes the term "narrow intelligence" somewhat of a misnomer, since it suggests some level of similarity or even continuity between "narrow" and general intelligences. In my model, this is not the case: demons and non-demons are different species—full stop. If you think this is a relevant crux, we can try and double-click on some of these concepts to expand them.

and how general do you think a large language model is?

I think this question is basically answered above, in the first section about demons? To reiterate: I think the Transformer architecture is definitely capable of playing host to a demon—but this isn't actually a concession in any strong sense, in my view: any Turing-complete architecture can host a demon, and the relevant question is how easy it is to summon a demon into that architecture. And again, currently I don't see the training and structure of Transformer-based LLMs as conducive to demon-summoning.

This is an interesting question on which I've gone back and forth.  I think ultimately, the inability to recognize blatant inconsistencies or to reason at all means that LLMs so far are not intelligent. (Or at least not more intelligent than a parrot.)