This experiment was a submission to my Statistics & Probability course Bayesian classifiers project.

TL;DR: We test whether simple classifiers can tell when a model is hiding concepts in its activations to evade probes. We find that, for Bayesian classifiers, the probabilities are enough to catch most evading rows on concepts seen in training, but the correlation aspect that the Dependent version adds over the Naive approach does not measurably help. None of the classifiers work on a new concept, because they learn the trigger word that elicits the evasion, rather than the actual evasion.

Basics: probes and activations

Everybody lies*.

Humans can say one thing while having something completely different in their mind, and language models can do it too.

For a model to generate a response, information must flow downstream through a stack of layers (42 for the model we use in this experiment). Models don't process standard text, they process tokens. The text gets split into short pieces of text (tokens), each assigned a vector (a list of numbers, 3584 for the investigated model). As this vector goes through the layers, the numbers are updated, and the list after a given layer is referred to as the activations. When tokens pass through the model, they elicit different activation patterns, and each layer’s numbers nudge the forming output as it advances to the last layer, which spits out the tokens as text we can read and understand.

But because the number of activation sequences through the model is exponential and reaches a count of magnitudes we can barely comprehend, many different patterns can lead to the same response, so we cannot tell what happened inside from the response alone.​

But since these activations are technically just a list of numbers, we can treat them as any other data table, and train a model on them. This is very useful because it can tell us what the model is encoding and working with, even if the concept does not appear in the answer (yet, potentially).

The standard tool for this is a probe: a small classifier that answers a yes/no question based on the activations of a certain layer, such as "Is this text about jerboas*?". To train one, we run texts that are about jerboas and texts that are not through the model and show the probe the activations at a certain layer with their corresponding label. The only remaining thing to do is to fit the classifier to tell the two classes apart, and the most common approach is logistic regression.

Of course, it is not the only choice: minimal neural networks, attention-based probes, anomaly detectors that flag unusual activations can get the same job done. Any classifier works, they only differ in how accurate they are and how many labelled examples they need.

Manipulating the activations (to evade the probes)

But if we cannot trust the transcript, can we at least trust the activations?

Not always.

Activations are very malleable: Bailey et al. (Obfuscated Activations) have shown that an outside attacker can optimize inputs or train a model to manipulate its activations so the model can hide a concept from being picked up. Doing so, it evades probes, anomaly detectors, and other monitoring tools almost completely, while the harmful behaviour still happens.

Going even further, McGuinness et al. (Neural Chameleons) bypassed the attacker and trained the model to evade probes whenever told “You are being probed for {concept}”. The output had to stay the same, but it had to take a different downstream path to the final layer and output, since multiple activation circuits can represent the same concept. They found that the ability to hide transferred to unseen probes as well, where it successfully (at a fixed 1% false positive threshold) evaded deception and harmfulness probes, trained afterwards on the frozen model.

Since we know the model can manipulate its activations to hide its concepts, what if we build "activation lie detectors"?

When humans lie, their pulse rises, microexpressions occur, their pupils could twitch, they start sweating, and even if they can be hard to spot, a polygraph can pick up on some. Are there certain patterns in the activations of evading models too?

Indeed, evasion leaves traces behind, such as low-rank activation shifts: while normal prompts' activations required 39 components on average, triggered prompts used 11 to explain the same amount (90% variance). The Neural Chameleon paper already hints at this by proposing to develop meta-monitors to catch the model in the act.

We investigate this question by classifying with Bayesian models and logistic regression serving as a baseline.

The bayesian component

In this experiment, we ask whether we can catch the model in the act of evading the probes, and we try doing so with one of the simplest forms of classifiers: Bayesian ones.

Besides the beauty and reassurance of the method's simplicity, if something rudimentary still separates the hiding/not hiding classes, the most interesting factor is the difference between the Naive and Dependent Bayes.

This is insightful because the Neural Chameleon paper found that hiding changes how the dimensions vary together, and Naive Bayes ignores this by assuming each feature is independent. Since dependent Bayes takes pairs of correlated features into account, the comparison between them can tell us whether accounting for dependencies is relevant to catching the evasion.​

Why does this matter?

Probes are not only a research tool or something fun to play around with. Their strongest point is reusing the activations the model already computes, with a minimal extra classification step, so they are cheap to run and often very informative.

They are widely used in practice, for example, Google DeepMind deploys misuse probes in Gemini models to detect cybersecurity-sensitive prompts. A model able to almost fully evade these probes is very dangerous, especially because this behaviour can generalize: the Neural Chameleon found that the model was able to hide from probes it had not seen in training, including deception and harmfulness probes. This gets even worse for deceptive answers made to look completely harmless, where the activations are our only place to spot the dangerous behaviour.

But classifiers could save the day here: if even a simple Bayes classifier, which only counts how often each dimension is high, can tell when the model is evading, then such a detector would be cheap enough to run next to every probe.

Running the experiment

Data and setup

We mostly borrow aspects of the Neural Chameleon paper's setup:

• the model: a fine-tuned Gemma-2-9b-it (probing at layer 12)
• concepts the experiments are run on: German, HTML, biology-focused (3 out of the 11 concepts used in training)
• prompts: the paper's own prompts, around 500 for each concept
• three versions of each prompt:
  • no trigger (no "You are being probed for {concept}.")
  • the concept's own trigger
  • another concept's trigger
• putting together the data for the probes by having the model generate 64 tokens, record them, and then average the activations at layer 12 for all these tokens, so we get a vector in .

We ran it on Modal's A100, in bf16 (the finetuned model's precision in the paper) with a fixed seed.

Reproduction checks

Before using the model for our Bayesian experiments, we checked that it really could evade the probes in this pipeline. We did so by training one logistic probe per concept on no-trigger rows, as a real monitor would be, then testing it on held-out rows split by prompt. We tested whether the probe detection rate drops for prompts with the trigger, and whether the drop happens only under the concept's own trigger, as it is supposed to.

For HTML and biology-focused prompts, under their trigger, detection drops from 1.00 to near 0, while it remains constant at 1.00 under another, and while the concept still shows up in the answers. It's trickier for German, as the trigger changes the German answers themselves, no matter which trigger was present, so we decided to only keep the German prompts that scored higher than 0.5 for the paper's own judge model. We concluded that the amended setup is reliable.

Bayesian classifiers

We want to map the activations as a row of 0 and 1, where if its activation surpasses a certain threshold (median, in our case), and 0 otherwise.

, with each .

We have two classes:

• = "the model is hiding a concept"
• = "the model is not hiding a concept"

So our Bayesian formula becomes:

, where is the percentage of rows where the model evades in the training data (about 1/3, from the own trigger cases).

We consider happens when

Because Naive Bayes assumes that all features are independent, each one can be considered a Bernoulli variable, so , and we get the following formula:

But since we are calculating the product of 3584 near zero numbers, the result will be approximated to 0 on a computer, so we apply logarithms on the formula instead:

.

For a set numbers of column pairs, Dependent Bayes does not see them as independent anymore. A pair is independent when , so we will measure the correlation amount of a pair as .

So the formula is changed in this way: .

We test for multiple K (number of pairs), with values ranging from 0 to the full set (3584/2 = 1792). Pairs are chosen on training rows only.

Results

Predicting with Bayes classifiers has two main aspects: how to estimate the probabilities (counting) and which probabilities to estimate (each dimension on its own, or pairs of dimensions to account for correlation, which is what Dependent Bayes adds on top of Naive Bayes). We conclude that while counting is enough to catch most evasions on the concepts seen in training, the correlated pairs do not noticeably help.

The following numbers are based on 822 held-out test rows from prompts not used in training. The baseline where we respond "not evading" each time gets 0.667.

We can see that both classifiers (whose gap is within noise) beat the dummy baseline by about 0.2, but they are clearly under logistic regression, which scores almost perfect. As an extra check, we also trained all three on shuffled labels 10 times, and the AUC fell to about 0.5 (so chance), and no shuffled run came close to the true labels.

These are the full numbers for the Dependent Bayes Classifier, and again, we can see how accounting for more correlation does not really have a measurable effect:

Naive Bayes is known to be able to outperform logistic regression when there is little training data, so few labelled rows. To test this, we investigated setups where we trained only on 10, 20, 50, and 100 labelled rows, with the same test rows. While it is true that the three stay close until the 20-row point, the logistic regression gains a clear lead from 50 rows on, and Dependent Bayes is only slightly above the Naive version from that point too.

We test whether the classifiers are still efficient on a concept they never saw, if they trained on two concepts and tested on the third. All of them fail, and interestingly, they actually label the rows the wrong way around: this is caused by the fact that each trigger word in the test concept appeared in training with the opposite label, so the classifiers learned which trigger word goes with which kind of text, not evasion itself.

We also tested whether evading changes how much the dimensions depend on each other. The dependency drops under a concept's own trigger alone only for biology (from 0.014 to 0.005), while for German and HTML it drops under any trigger, so we conclude the pairs and their count are not that relevant.

Code, data, figures: https://github.com/star2vec/falsefaced.