Tl;dr: There should be a clear, well-designed eval on activation faithfulness: whether the model activations a monitor reads remain a trustworthy surface when the model suspects/knows it is being monitored. We provide an appendix with preliminary results towards such an eval in case people want to build off of it, but we think way more work needs to be done here.
—
There is currently a lot of discussion around the precarious state of CoT monitorability. To our knowledge, CoT monitorability is threatened by at least four things:
Although it is emphatically clear that CoT monitorability is valuable. But it is also important to have a defense-in-depth posture. To this end, an additional monitoring tool is the activation monitor. On first glance, activation monitors might be helpful for addressing points 1, 2 and 4 above. However, activation monitors come with their own challenges. For instance:[1]
It’s hard to tell whether the activation monitor is well-calibrated to the underlying behavior.
Even if the activation monitor is well-calibrated, it is not clear whether a sufficiently capable model can, in-context, understand its activations are being monitored and control its activations to dodge the monitor.
This post concerns the second point: models deliberately dodging activation monitors. We think it is reasonable to expect that this will happen. For instance, suppose a highly persistent model is working on a task and its activations are being monitored by a well-calibrated monitor. This highly persistent model keeps getting its attempts flagged by the monitor. Then the highly persistent model’s primary recourse is to keep trying to modify its own context in an attempt to dodge the monitor. It is particularly concerning if models are RL-ed (or otherwise trained) against monitors where they might learn this behavior, but we also think it might happen without any training.
Given the above, we think it is important to have a principled eval for tracking activation faithfulness.
There has been work on this already, including this post and this paper. We also include some experiments from an ERA project in the appendix of this post. However, the current work seems to fall short in several ways:
To our knowledge, there isn’t a go-to eval for models’ ability to evade activation probes in-context.
Ideally, such an eval should work across model architectures, probe topics and usage contexts.
It’s not obviously clear to us how j-space vs. regular probes should interact for the purposes of this eval.
Such an eval should prioritize large models! For many reasons, we suspect that negative results on small models do not imply negative results on large ones.
Most of the literature is on models’ ability to attend to their activations. This likely carries many implications for their ability to control these activations, but it’s not clear how this maps out.
Labs seem to love optimizing leaderboards! It therefore seems good to set up a leaderboard for models’ activation faithfulness and maximally require labs to submit to this eval.
If the models are able to evade the probes, it would be great to catalog the ways in which this happens. For instance, models might evade a deception probe by (a) no longer being deceptive or (b) being even more deceptive. The former is clearly better than the latter and we should understand the techniques the models have available to them.
If you'd like to work on this (or are already doing so), feel free to reach out as it would be good to pool efforts.
—
We provide all code here and experiments in an appendix via this Google Doc.
Tl;dr: There should be a clear, well-designed eval on activation faithfulness: whether the model activations a monitor reads remain a trustworthy surface when the model suspects/knows it is being monitored. We provide an appendix with preliminary results towards such an eval in case people want to build off of it, but we think way more work needs to be done here.
—
There is currently a lot of discussion around the precarious state of CoT monitorability. To our knowledge, CoT monitorability is threatened by at least four things:
Although it is emphatically clear that CoT monitorability is valuable. But it is also important to have a defense-in-depth posture. To this end, an additional monitoring tool is the activation monitor. On first glance, activation monitors might be helpful for addressing points 1, 2 and 4 above. However, activation monitors come with their own challenges. For instance:[1]
This post concerns the second point: models deliberately dodging activation monitors. We think it is reasonable to expect that this will happen. For instance, suppose a highly persistent model is working on a task and its activations are being monitored by a well-calibrated monitor. This highly persistent model keeps getting its attempts flagged by the monitor. Then the highly persistent model’s primary recourse is to keep trying to modify its own context in an attempt to dodge the monitor. It is particularly concerning if models are RL-ed (or otherwise trained) against monitors where they might learn this behavior, but we also think it might happen without any training.
Given the above, we think it is important to have a principled eval for tracking activation faithfulness.
There has been work on this already, including this post and this paper. We also include some experiments from an ERA project in the appendix of this post. However, the current work seems to fall short in several ways:
If you'd like to work on this (or are already doing so), feel free to reach out as it would be good to pool efforts.
—
We provide all code here and experiments in an appendix via this Google Doc.
To be clear, this is an inexhaustive list.