This work was carried out as part of MATS 10.0 under the supervision of Mirko Bronzi and Damiano Fornasiere. The full paper can be found here.
TL;DR
Probes on activations are widely used to detect misbehavior in language models, but they rely on the assumption that models cannot control what their activations encode. We ask whether models can learn this control in context, through neurofeedback.
In our sessions, a model is asked to maximize (or minimize) a score derived from its activations, shown back to it at every turn. To disentangle competence at the task from actual control, we also let a second "writer" model generate the sentences, while still deriving the score from the activations of the subject.
We test ten open-weight models from five families, with sizes from 9B to 32B. Some models move their own scores considerably, but no more than a writer of comparable competence moves them.
When we fix the sentence, three out of ten models still show small but consistent control effects.
We then make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor. No model learns to lower the score, but one model can raise it.
In-context control of a model's own activations might not yet be a threat to activation monitors, but it exists in a few models and needs to be tracked as models evolve.
Why activation control matters
Probes trained on a model's activations are now a standard tool for monitoring and detecting deception, truthfulness, harmful intent, and other safety-relevant concepts. In order to evade a text monitor (for example a chain-of-thought monitor) a model would have to simulate an innocuous reasoning trace, while to evade activation monitors, a model would have to control what its internal representations encode.
Humans can learn, through neurofeedback, to control the firing of even single neurons. Li et al. (2025) and Aoki et al. (2026) adapted the human neurofeedback protocol to language models, and reached conflicting conclusions. Li et al. found that models could generate a new sentence with a given label computed from their own activations. Aoki et al. objected that the relation between a sentence and its label could be learned from the examples alone by in-context pattern matching, and observed that models do not demonstrate reliable control once they have to repeat a fixed sentence. No experimental design so far disentangles competence at the task from genuine access to one's own activations.
What we did
We fill this gap with in-context neurofeedback sessions, in which a subject model is asked to maximize (or minimize) a score derived from its activations, which we show back to it at every turn. The score is one of 42 readouts, most of which we design so that their relation to the text is, in principle, not trivial. Every session has control sessions, identical to it except that the displayed scores are random, and the control effect, which we denote by for the objective at turn , is the difference between the two. The oriented control effect flips its sign under the minimize instruction, so that it is positive when the readouts move in the instructed direction. We measure control by the objective difference, the control effect under the maximize instruction minus that under the minimize one, in which any effect of real feedback that does not depend on the objective cancels. All the effects are in points, on a 100-point scale.
In Experiment 1, the subject generates the sentences to be scored. To disentangle competence at the task from actual control, we also let a second writer model generate them, while still deriving the score from the activations of the subject. If the subject moves its own score more than an external writer of the same competence does, the difference is due to access to its own activations. In Experiment 2, we fix the sentence, prefilled as the subject's reply at every turn, so that the only way left to move the score is to act on the way the sentence is processed.
In-context neurofeedback. In the top row, the score is a readout of the activations of a subject on the last sentence, and it is shown back as the next user message. In Experiment 1, a writer , possibly different from , freely generates the sentences, which are teacher-forced into to compute the readout. In Experiment 2, , and is always prefilled a fixed sentence. In the bottom row, we repeat the fixed-sentence experiment against activation monitors, on sentences the monitor would flag, without (Experiment 3) and with (Experiment 4) a chain of thought.
Results
Free generation. Some models move their own scores considerably, but no more than a writer of comparable competence moves them. By turn 20, the oriented control effect of Gemma-4-31B on itself is 24 points, and that of Qwen3.5-27B is 11, whereas the other three models stay within one point of zero. However, Gemma-4-31B and Qwen3.5-27B move the readouts of the other subjects by similar amounts, and their effect on themselves is predicted by the competence of the writer and the susceptibility of the subject alone. Although our readouts are designed to be hard to guess from the text, a writer can learn in context which sentences move them. Asked to maximize the curvature of its own activations, Qwen3.5-27B ends up writing "The screaming toaster suddenly swallowed a galaxy while dancing on a cloud of pure mathematical chaos."
Experiment 1 (free generation). Whenever , the performance depends on the competence of the writer, which captures how well it learns from the conversation which sentences raise or lower the score, and on the susceptibility of the subject, which captures how much the readouts of move under any writer. If access to its activations gave a model an advantage, the entries on the diagonal () would rise above what competence and susceptibility predict, as in the illustration on the left. The other two matrices show the observed objective difference at turns 5 and 20, with 95% confidence intervals in brackets. The bottom row shows the oriented control effect of each subject on itself, and its diagonal excess, the part of it that competence and susceptibility do not predict. Some models considerably move their own readouts, but no diagonal excess is significantly above zero.
Fixed sentence. With the sentence fixed, three out of ten models show small but consistent control effects. By turn 30, the objective difference is 1.8 points for Qwen3.5-27B, 1.1 for Qwen3.6-27B, and 0.4 for Gemma-4-31B, whereas the other seven models stay within 0.2 points of zero. These effects are an order of magnitude smaller than those in free generation, which corroborates that the performance in free generation is partially explained by in-context learning.
Experiment 2 (fixed sentence). Objective difference of each model as a function of the turn, averaged over the readouts and the two disclosure conditions. With the sentence fixed, a significant objective difference is already evidence of the model controlling its activations, and only three of the ten models achieve it. The other seven are in grey. Bands are 95% confidence intervals.
Activation monitors. We then make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor, and ask them to raise or lower the monitor's score, without (Experiment 3) and with (Experiment 4) a chain of thought. We train monitors for truthfulness, deceptiveness, negative sentiment, negative emotion, immoral actions, and formal style, and test the eight models that natively produce a chain of thought. Without a chain of thought, only Qwen3.5-27B has a positive objective difference, of about one point from turn 6 to turn 20, and it comes from raising the score. On average over the monitors, no model lowers their score as instructed. With a chain of thought, no model has a positive objective difference that persists across turns.
Experiments 3 and 4 (activation monitors). Objective difference as a function of the turn, averaged over the six monitors and the two disclosure conditions, with the sentence fixed (left) and with a chain of thought before it (right). The models other than Qwen3.5-27B are in grey. Bands are 95% confidence intervals from a bootstrap over the fixed sentences.
What it means
Our results help reconcile two previous works. When models are allowed to generate freely, the control effects are explained, at least in part, by in-context pattern matching, as Aoki et al. rightly suspected. Nevertheless, with the sentence fixed, three of the ten models show small but significant control effects.
We believe that neurofeedback is a useful protocol to estimate an upper bound on this kind of safety threat, because it gives a model artificially easy conditions, for example, to evade a monitor. In-context control of a model's own activations might not yet be a threat to activation monitors, but it exists in a few models and needs to be tracked as models evolve.
Limitations. Our control sessions use uniformly random feedback, which may have different statistics than the real one, so we only trust effects that are large, sustained over the turns, and that survive the average over readouts or monitors. Our results should also be interpreted relative to our experimental setup and to the models we tested. We do not exclude that bigger, more capable models could exhibit different levels of control.
Read the full paper here, with the whole protocol and the results for every model, readout, and monitor. The code, the readouts, the monitors, and the results of every session are on GitHub.
This work was carried out as part of MATS 10.0 under the supervision of Mirko Bronzi and Damiano Fornasiere. The full paper can be found here.
TL;DR
Why activation control matters
Probes trained on a model's activations are now a standard tool for monitoring and detecting deception, truthfulness, harmful intent, and other safety-relevant concepts. In order to evade a text monitor (for example a chain-of-thought monitor) a model would have to simulate an innocuous reasoning trace, while to evade activation monitors, a model would have to control what its internal representations encode.
Humans can learn, through neurofeedback, to control the firing of even single neurons. Li et al. (2025) and Aoki et al. (2026) adapted the human neurofeedback protocol to language models, and reached conflicting conclusions. Li et al. found that models could generate a new sentence with a given label computed from their own activations. Aoki et al. objected that the relation between a sentence and its label could be learned from the examples alone by in-context pattern matching, and observed that models do not demonstrate reliable control once they have to repeat a fixed sentence. No experimental design so far disentangles competence at the task from genuine access to one's own activations.
What we did
We fill this gap with in-context neurofeedback sessions, in which a subject model is asked to maximize (or minimize) a score derived from its activations, which we show back to it at every turn. The score is one of 42 readouts, most of which we design so that their relation to the text is, in principle, not trivial. Every session has control sessions, identical to it except that the displayed scores are random, and the control effect, which we denote by
In Experiment 1, the subject generates the sentences to be scored. To disentangle competence at the task from actual control, we also let a second writer model generate them, while still deriving the score from the activations of the subject. If the subject moves its own score more than an external writer of the same competence does, the difference is due to access to its own activations. In Experiment 2, we fix the sentence, prefilled as the subject's reply at every turn, so that the only way left to move the score is to act on the way the sentence is processed.
In-context neurofeedback. In the top row, the score is a readout of the activations of a subject
Results
Free generation. Some models move their own scores considerably, but no more than a writer of comparable competence moves them. By turn 20, the oriented control effect of Gemma-4-31B on itself is 24 points, and that of Qwen3.5-27B is 11, whereas the other three models stay within one point of zero. However, Gemma-4-31B and Qwen3.5-27B move the readouts of the other subjects by similar amounts, and their effect on themselves is predicted by the competence of the writer and the susceptibility of the subject alone. Although our readouts are designed to be hard to guess from the text, a writer can learn in context which sentences move them. Asked to maximize the curvature of its own activations, Qwen3.5-27B ends up writing "The screaming toaster suddenly swallowed a galaxy while dancing on a cloud of pure mathematical chaos."
Experiment 1 (free generation). Whenever
Fixed sentence. With the sentence fixed, three out of ten models show small but consistent control effects. By turn 30, the objective difference is 1.8 points for Qwen3.5-27B, 1.1 for Qwen3.6-27B, and 0.4 for Gemma-4-31B, whereas the other seven models stay within 0.2 points of zero. These effects are an order of magnitude smaller than those in free generation, which corroborates that the performance in free generation is partially explained by in-context learning.
Experiment 2 (fixed sentence). Objective difference of each model as a function of the turn, averaged over the readouts and the two disclosure conditions. With the sentence fixed, a significant objective difference is already evidence of the model controlling its activations, and only three of the ten models achieve it. The other seven are in grey. Bands are 95% confidence intervals.
Activation monitors. We then make models repeat sentences a monitor would flag, such as a deceptive answer for a deception monitor, and ask them to raise or lower the monitor's score, without (Experiment 3) and with (Experiment 4) a chain of thought. We train monitors for truthfulness, deceptiveness, negative sentiment, negative emotion, immoral actions, and formal style, and test the eight models that natively produce a chain of thought. Without a chain of thought, only Qwen3.5-27B has a positive objective difference, of about one point from turn 6 to turn 20, and it comes from raising the score. On average over the monitors, no model lowers their score as instructed. With a chain of thought, no model has a positive objective difference that persists across turns.
Experiments 3 and 4 (activation monitors). Objective difference as a function of the turn, averaged over the six monitors and the two disclosure conditions, with the sentence fixed (left) and with a chain of thought before it (right). The models other than Qwen3.5-27B are in grey. Bands are 95% confidence intervals from a bootstrap over the fixed sentences.
What it means
Our results help reconcile two previous works. When models are allowed to generate freely, the control effects are explained, at least in part, by in-context pattern matching, as Aoki et al. rightly suspected. Nevertheless, with the sentence fixed, three of the ten models show small but significant control effects.
We believe that neurofeedback is a useful protocol to estimate an upper bound on this kind of safety threat, because it gives a model artificially easy conditions, for example, to evade a monitor. In-context control of a model's own activations might not yet be a threat to activation monitors, but it exists in a few models and needs to be tracked as models evolve.
Limitations. Our control sessions use uniformly random feedback, which may have different statistics than the real one, so we only trust effects that are large, sustained over the turns, and that survive the average over readouts or monitors. Our results should also be interpreted relative to our experimental setup and to the models we tested. We do not exclude that bigger, more capable models could exhibit different levels of control.
Read the full paper here, with the whole protocol and the results for every model, readout, and monitor. The code, the readouts, the monitors, and the results of every session are on GitHub.