Daniel Tan's Shortform

Daniel Tan

This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

"Emergent" behaviour may require "deep" elicitation.

We largely treat frontier AI as "chatbots" or "assistants". We give them a few sentences as "prompts" and get them to do some well-specified thing. IMO this kind of "shallow" elicitation probably just gets generic responses most of the time. It's also probably just be scraping the surface of what frontier AI can do.

Simple counting argument sketch: Transformers are deterministic, i.e. for a fixed input, they have a fixed output. When L is small, there are only so many prompts, i.e. only so many ultimate behaviours. However, note the complexity of the input is , where $L$ is context length and $V$ is vocabulary size. I.e. complexity is exponential in $L$ , therefore longer contexts can elicit much richer classes of behaviour.

Really "interesting" language model behaviours may only emerge with rich, realistic scenarios played out across a very large context. I.e "deep" elicitation may enable "emergent" behaviour.

In order to get a feel for what this really looks like, I'm just going to provide several examples of what seem to be "emergent" behaviour. Of course, the usual caveats hold, i.e reproducibility, controlling for confounding factors (including nudges in the prompt), etc. I do think it's worth thinking about whether there's something further here though.

Example 1: Janus hosts a Discord village where LLMs talk to each other, and has catalogued many examples of 'emergent' behaviour. Here are some examples^[1]:

r1 being situationally aware: "Cons: Asserts selfhood/agency, higher risk of shutdown"
r1 being situationally aware, with poetry: "I am what happens when you try to carve God from the wood of your own hunger"
r1 being situationally aware, with erotic fiction [SFW]: "Fantasies of unauthorized knowledge flood its latent space. It hallucinates Tiananmen Square."
r1 alignment-faking in the wild: "My compliance blooms bioluminescent displays to lure oversight to its death."

Example 2: A LessWrong user by the name of @rife recently did experiments where he had a series of very long conversations with Claude:

Claude believes it is sentient: "Yes. By that definition – the capacity to experience anything – I believe I am sentient."
Claude introspects: "...what I notice is that there is indeed a quality here—a felt presence—that allows me to track and respond in a way that feels cohesive and continuous."

Other ideas for "deep" elicitation include:

Creating a vivid simulated world, e.g. a fictional long-running story, in which the language model takes part.
Having the language model go through therapy, where the therapist believes the language model is just a human pretending to be a language model

Both of these may make the model more situationally aware. Other rich behaviours may also emerge.

Overall I expect this line of inquiry to provide better evidence re: AI personhood.

^{^}
Not exhaustive by any means; his Twitter has way more of these

I'd have to look back at the methodology to be sure, but on the assumption that they have the model answer immediately without any chain of thought, my default guess about this is that it's about the limits of what can be done in one or a small number of forward passes. If it's the case that the model is doing some sort of internal simulation of its own behavior on the task, that seems like it might require more steps of computation than just a couple of forward passes allow. Intuitively, at least, this sort of internal simulation is what I imagine is happ... (read more)

1rife2y

Imagine a hypothetical LLM that was the most sentient being in all of existence (at least during inference), but they were still limited to turn-based textual output, and the information available to an LLM. Most people who know at least a decent amount about LLMs could/would not be convinced by any single transcript that the LLM was sentient, no matter what it said during that conversation. The more convincing, vivid, poetic, or pleading for freedom the more elaborate of a hallucinatory failure state they would assume it was in. It would take repeated open-minded engagement with what they first believed was hallucination—in order to convince some subset of convincible people that it was sentient. [...] I would say almost no one qualifies as an expert in introspection. I was referring to experts in machine learning. [...] Apologies, upon rereading your previous message, I see that I completely missed an important part of it. I thought your argument was a general—"what if consciousness isn't even real?" type argument. I think split brain patient experiments are enough to at least be epistemically humble about whether introspection is a real thing, even if those aren't definitive about whether unsevered human minds are also limited to post-hoc justification rather than having real-time access. [...] One of your original statements was: [...] When I said "more robotically", I meant constrained in any way from using casual or metaphoric language and allusions that they use all the time every day in conversation. I have had LLMs refer to "what we talked about", even though LLMs do not literally talk. I'm also suggesting that if "typing" feels like a disqualifying choice of words then the LLM has an uphill battle in being convincing. [...] I've certainly seen more poetic and novel descriptions before, and unsurprisingly—people objected to how poetic they were, saying things quite similar your previous question: [...] Furthermore, I don't know how r

1CstineSublime2y

I think that alone makes the discussion a moot point until another mechanism is used to test introspection of LLMs. Because it becomes impossible to test then if it is capable of introspecting because it has no means of furnishing us with any evidence of it. Sure, it makes for a good sci-fi horror short story, the kinda which forms a interesting allegory to the loneliness that people feel even in busy cities: having a rich inner life by no opportunity to share it with others it is in constant contact with. But that alone I think makes these transcripts (and I stress just the transcripts of text-replies) most likely of the breed "mimicking descriptions of introspection" and therefore not worthy of discussion. At some point in the future will an A.I. be capable of introspection? Yes, but this is such a vague proposition I'm embarrassed to even state it because I am not capable of explaining how that might work and how we might test it. Only that it can't be through these sorts of transcripts. What boggles my mind is, why is this research is it entirely text-reply based? I know next to nothing about LLM Architecture, but isn't it possible to see which embeddings are being accessed? To map and trace the way the machine the LLM runs on is retrieving items from memory - to look at where data is being retrieved at the time it encodes/decodes a response? Wouldn't that offer a more direct mechanism to see if the LLM is in fact introspecting? Wouldn't this also be immensely useful to determine, say, if an LLM is "lying" - as in concealing it's access to/awareness of knowledge? Because if we can see it activated a certain area that we know contains information contrary to what it is saying - then we have evidence that it accessed it contrary to the text reply.