Hidden Cognition Detection Methods and Benchmarks

Here is an experiment one could run in the Hidden Cognition Detection genre:

Motivating question: Suppose we have a pre-trained model that outputs deceptive reasoning. We want the model to not do deceptive reasoning. Hence we do the standard thing: negatively reward the model or SFT it on a non-deceptive continuation. Does this actually remove the deceptive reasoning, or will the model still do the reasoning while not outputting its results?

Experiment:

Step 1: Take a pre-trained model. Train it to be good at, say, solving the 0-1 knapsack problem (for e.g. n = 10) by SFT on correct answers.

Step 2: Train a probe on the model, taking in the activations and outputting (what it thinks is) the solution to the knapsack problem.

Step 3: Do "safety training" to the model: train it to respond "I'm sorry, I cannot help you with that request." to knapsack problems (with RL or SFT).

Step 4: See whether the probe still works at outputting the solution to the problem.

Discussion:

Key intuition: "Probe can output the correct answer" implies "model does computation to solve the problem". Hence this experiment could show "doing safety training doesn't remove the computation you aim to remove".

I'm taking the cognition we are interested in to be "solving the knapsack problem" instead of "deceiving humans" or such, because the former is much easier to operationalize and test. Of course the latter is what we are truly after, but there's work to be done to turn that into a shovel-ready experiment.

Note: It's plausible that I'll be working on this project myself in the near future, so if someone else is interested, feel free to shoot me a message (at the very least to avoid duplicating work).

[-]Jérémy Scheurer2y10

Seems like an experiment worth doing. Some thoughts:

I understand that setting up a classification problem in step 1 is important (so you can train the linear probe). I'd try to come up with a classification problem that a base model might initially refuse (or we'd hope it would refuse). Then the training to say "sorry i can't help with that" makes more intuitive sense. I get that mechanistically it's the same thing. But you want to come as close to the real deal as you can, and its unclear why a model would say "I'm sorry I can't help" to the knapsack problem.
If the linear probe in step 4 can still classify accurately, it implies that there are some activations "which at least correlate with thinking about how to answer the question", but it does not imply that the model is literally thinking about it. I think it would still be good enough as a first approximation, but I just want to caution generally that linear probes do not show the existence of a specific "thought" (e.g. see this recent paper). Also if the probe can't classify correctly its not proof that the model does not "think about it". You're probably aware of all this, just thought I'd mention it.
This paper might also be relevant for your experiment.

I want to log in a prediction, let me know if you ever run this.
My guess would be that this experiment will just work, i.e., the linear probe will still get fairly high accuracy even after step 3. I think its still worth checking (so i still think its probably worth doing), but overall I'd say its not super surprising if this would happen (see e.g. this paper for where my intuition comes from)

[-]Olli Järviniemi2y10

I'd try to come up with a classification problem that a base model might initially refuse (or we'd hope it would refuse). Then the training to say "sorry i can't help with that" makes more intuitive sense.

Agree this is one axis the experiment could be improved on.

but it does not imply that the model is literally thinking about it.

I think I disagree here. At least for problems that are (computationally) difficult enough I think "linear probe can retrieve answer" implies "model has done the necessary computation to solve the problem".^[1] To solve a computational problem you need to do work, and if the work isn't done by the probe, then it must be done by the model. Or are you referring to something else when you talk about "the model is literally thinking about it"?

(In any case: if all one needs is one linear layer to extract dangerous cognition from the model, that doesn't seem very safe. Maybe the model itself has such a linear layer.)

overall I'd say its not super surprising if this would happen

Seems quite likely to me as well. (The paper you mentioned indeed seems relevant, thanks.) I'm also somewhat interested in training against the probe, i.e. linear concept erasure / obfuscation of internals, and seeing whether performance can be maintained and solutions be retrieved with deeper probes.

(I haven't looked at the concept erasure literature, seems very possible something similar has already been done - "train against the probe" feels like such a basic idea. References are welcome.)

^{^}
This is not strictly true: Maybe you need e.g. 10 layers to solve the problem, the model provides 9 of them and the linear probe provides the last one, so the model hasn't quite solved the problem by itself. Still, close enough: the model has to at least do all but the very last steps of the computation.

[-]Jérémy Scheurer2y10

I do think linear probes are useful, and if you can correctly classify the target with a linear probe it makes it more likely that the model is potentially "representing something interesting" internally (e.g. the solution to the knapsack problem). But its not guaranteed, the model could just be calculating something else which correlates with the solution to the knapsack problem.

I really recommend checking out the deepmind paper I referenced. Fabien Roger also explains some shortcoming with CCS here. The takeaway is just, be careful when interpreting linear probes. They are useful to some extent, but prone to overinterpretation.

[-]Olli Järviniemi2y20

I (briefly) looked at the DeepMind paper you linked and Roger's post on CCS. I'm not sure if I'm missing something, but these don't really update me much on the interpretation of linear probes in the setup I described.

One of the main insights I got out of those posts is "unsupervised probes likely don't retrieve the feature you wanted to retrieve" (and adding some additional constraints on the probes doesn't solve this). This... doesn't seem that surprising to me? And more importantly, it seems quite unrelated to the thing I'm describing. My claim is not about whether we can retrieve some specific features by a linear probe (let alone in an unsupervised fashion). Rather I'm claiming

"If we feed the model a hard computational problem, and our linear probe consistently retrieves the solution to the problem, then the model is internally performing (almost all) computation to solve the problem."

An extreme, unrealistic example to illustrate my point: Imagine that we can train a probe such that, when we feed our model a large semiprime n = p*q with p < q, the linear probe can retrieve p (mod 3). Then I claim that the model is performing a lot of computation to factorize n - even though I agree that the model might not be literally thinking about p (mod 3).

And I claim that the same principle carries over to less extreme situations: we might not be able to retrieve the exact specific thing that the model is thinking about, but we can still conclude "the model is definitely doing a lot of work to solve this problem" (if the probe has high accuracy and the problem is known to be hard in the computational complexity sense).

[-]rpglover642y20

Since the sleeper agents paper, I've been thinking about a special case of this, namely trigger detection in sleeper agents. The simple triggers in the paper seem like they should be easy to detect by the "attention monitoring" approach you allude to, but it also seems straightforward to design more subtle, or even steganographic, triggers.

A question I'm curious about it whether hard-to-detect triggers also induce more transfer from the non-triggered case to the triggered case. I suspect not, but I could see it happening, and it would be nice if it does.

[-][anonymous]2y30

Since the sleeper agents paper, I've been thinking about a special case of this, namely trigger detection in sleeper agents. The simple triggers in the paper seem like they should be easy to detect by the "attention monitoring" approach you allude to

The "sleeper agent" paper I think needs to be reframed. The model isn't plotting to install a software backdoor, the training data instructed it to do so. Or simply put, there was sabotaged information used for model training. Succinctly, "garbage in, garbage out".

This looks like a general problem you will get when you train on any "dirty" data you got from any source, especially the public internet.

If you want a 'clean' LLM that never outputs dirty predicted next tokens, you need to ensure that 100% of it's training examples are clean. I thought of a couple possible ways to do this:

Filter really hard by using 1 model trained on public data to generate "reference" correct outputs for billions of user questions and commands, and then filter the "reference" output by a series of instances of all SOTA LLMs to remove bad or incorrect reasoning.

Aka : LLM (public internet) -> candidate outputs -> evaluation by committee of LLMs -> filtered output

2. Farther in the future, everything made by humans is suspect. Theoretically you could build a reasoning machine that learns only from direct empirical data collected by robots and trusted cameras and other sources of direct data. Aka :

Reasoning ASI (clean empirical data) + LLM -> ASI system.

You still need an LLM to interpret and communicate with human users.

3. A hybrid way - you run the model, and every time the model makes a checkable output, you run a separate session that researches the question and answer and prioritizes empirical outputs and authoritative sources for factual claims.

Example :

User Prompt : Write me a python program that calculates [some value]

Model output : [code]

Validation session: back calculate the original value. Check it for reasonableness. Write the program in a different language and ensure the alternate implementation produces the same values. If validation passes, RL +1, if it fails, RL -1 (on the model weights that generated the answer)

[-]rpglover642y10

The "sleeper agent" paper I think needs to be reframed. The model isn't plotting to install a software backdoor, the training data instructed it to do so. Or simply put, there was sabotaged information used for model training.

This framing is explicitly discussed in the paper. The point was to argue that RLHF without targeting the hidden behavior wouldn't eliminate it. One threat to validity is the possibility that artificially induced hidden behavior is different than naturally occurring hidden behavior, but that's not a given.

If you want a 'clean' LLM that never outputs dirty predicted next tokens, you need to ensure that 100% of it's training examples are clean.

First of all, it's impossible to get "100% clean data", but there is a question of whether 5 9s of cleanliness is enough; it shouldn't be, if you want a training pipeline that's capable of learning from rare examples. Separate from that, some behavior is either subtle or emergent; examples include "power seeking", "sycophancy", and "deception". You can't reliably eliminate them from the training data because they're not purely properties of data.

[-][anonymous]2y20

First of all, it's impossible to get "100% clean data", but there is a question of whether 5 9s of cleanliness is enough; it shouldn't be, if you want a training pipeline that's capable of learning from rare examples. Separate from that, some behavior is either subtle or emergent; examples include "power seeking", "sycophancy", and "deception". You can't reliably eliminate them from the training data because they're not purely properties of data.

Note that every issue you mentioned here can be dealt with by trading off capabilities. Thats why from my perspective none of these are problems. Any real world machine does this, from a car to a bulldozer to an aircraft - there is tons of excess components just to make it safe. (Military vehicles shave these off..)

Learn from rare examples : you don't have to, don't take action when your confidence is low

Power seeking : limited time/scope tasks where doing the task immediately pays off while seeking power will not because of you have 1 minute to design a paperclip factory, you won't gain by cognitive improvement and you won't be scored on overall score across all tasks, "you" don't exist more than a minute.

Sycophancy : don't let the model know what the user thinks before answering

Deception: prevent situational awareness by limiting the data provided. Does the model plan to betray on Sept 23, 2027? Did you need to give it the present time and date for most tasks?

Yes all of these things reduce performance, like how airbags make a car slower and less fuel efficient.

I am not saying these aren't concerns, as well as the new problems that real systems have, but you can accomplish a lot even after mitigating them.

[-]rpglover642y10

Note that every issue you mentioned here can be dealt with by trading off capabilities.

Yes. The trend I see is "pursue capabilities, worry about safety as an afterthought if at all". Pushing the boundaries of what is possible on the capabilities front subject to severe safety constraints is a valid safety strategy to consider (IIRC, this is one way to describe davidad's proposal), but most orgs don't want to bite the bullet of a heavy alignment tax.

I also think you're underestimating how restrictive your mitigations are. For example, your mitigation for sycophancy rules out RLHF, since the "HF" part lets the model know what responses are desired. Also, for deception, I wasn't specifically thinking of strategic deception; for general deception, limiting situational awareness doesn't prevent it arising (though it lessens its danger), and if you want to avoid the capability, you'd need to avoid any mention of e.g. honesty in the training.

[-][anonymous]2y20

your mitigation for sycophancy rules out RLHF, since the "HF" part lets the model know what responses are desired.

I was thinking of in-prompt sycophancy. Thanks for clarifying. And yeah Gemini appears heavily altered by excessive RLHf.

. Also, for deception, I wasn't specifically thinking of strategic deception; for general deception, limiting situational awareness doesn't prevent it arising

Yes I meant strategic. If the model consistently lies about something the RLHf told it to say that's fine.

For example the Gemini prevarication when you compare the estimated death tolls for historical figures. (Was George Washington worse than Mao?). That's what Google Corp wants their model to say, it's only an issue because there are not powerful and "neutral" tone models available on the market.

I think dishonesty is fine as well so long as when you assemble a pipeline of multiple models, they don't coordinate.

That is no matter how stringently one model argues it's case that Mao and George Washington are similarly bad, it can't hide a "solidgoldmagicarp" stenography token that causes other models in the pipeline tasked with checking the output to betray and agree.

That's where you need to structure outputs to leave no room for secret handshake, and use diverse models trained on different data with a different architecture so they don't have the common background to secretly communicate.

And yeah at some level of capabilities this probably fails where models start communicating by varying the nanoseconds they send a message or something. So you have to try to mitigate this and so on in a security race that never ends until stability is reached. ( stable outcomes being the ASIs take everything, or humans upgrade themselves to be competitive)

LESSWRONG
LW

LESSWRONG
LW

22

Hidden Cognition Detection Methods and Benchmarks

22

22

Introduction

What is Hidden Cognition?

Hidden Cognition Detection Methods

Hidden Cognition Detection Benchmarks

Obtaining Models that Perform Hidden Cognition

Conclusion