Epistemic status: I am new to AI Safety and am writing blogs to gain context. This blog post was formed from discussions with Aidan Ewart and Jonathan Bostock, but they do not necessarily endorse this post.
TL;DR
Given recent examples of AI misbehaviour during training episodes, AI companies might want to start using monitoring during training as well as deployment. But this might have the effect of training the AIs to simply evade the monitors. Depending on the specifics of the monitoring protocol, this evasion may be learned more or less quickly (or not at all). We refer to the time that it takes for an AI to learn to evade a monitoring setup as the “lifetime” of the monitoring setup, and make the case for investigating the factors which contribute to this lifetime.
It is probably true that frontier models currently behave, andwill behave, particularly badly in the training phase, as discussed in "Models may behave differently in graded episodes" (see “Everything we know suggests that the models in these incidents …”). This means that we should monitor RL rollouts carefully; we don’t want another huggingface-style incident (the next breakout may well be catastrophic).
However, we should be careful - strong (synchronous) monitoring in rollouts may teach the models to bypass the monitor. The more we rely on some monitor to flag malign behaviours during training, the stronger the optimisation pressure on the monitor is. So, we might face a trade off between monitorability and preventing our models taking catastrophic actions during training.
To get better trade offs, we want to find monitoring setups that are resistant to adversarial optimisation. I talk about two ways we can do this; changing the monitoring setup (see here), or changing the level of optimisation pressure we allow against the monitors (see here).
Changing the Monitoring Setup
In order to have a monitoring setup that is resistant to adversarial optimisation, we may have to make trade offs (for example, how sensitive a monitor is may affect the adversarial optimisation pressure on the monitor). Therefore, I suggest that it might be useful to investigate how quickly we might expect models to learn to bypass monitoring - that is, measure the lifetime of different monitoring methods. If results generalise, weighing up lifetime and accuracy could help inform us where we should aim to lie on this trade off:
For example, if method A is 99% accurate initially but drops to 50% accuracy after N many RL steps, we might prefer method B which is 98.5% accurate initially and stays above 95% accuracy for 10N RL steps.
Of course, how fine-grained the monitor is would also affect the lifetime of the monitor, but may not sufficiently incentivise the model to not reward hack. We talk more about this the next section.
There seem to be a lot of adaptable experimental setups for conducting these investigations; there is substantial work on looking at approaches which maximise monitoring accuracy, such as Ensemble monitoring for AI control and Combining Cost-Constrained Runtime Monitors, and these setups may be applicable to measuring the lifetimes of different monitoring protocols.
I think that we should consider using different monitoring methods in the RL rollouts and the evaluation runs. This would let us use a monitor that the model has not had an opportunity to train against. One more consideration here is how correlated the monitoring methods are - does the model learning to bypass monitor A mean that it can bypass monitor B? We can refer to this correlation as obfuscation transfer risk. (This is some great work which looks at this risk.)
I think that it would be a good investigation to look at what combinations of monitoring methods in the RL and evaluation stages minimise the proportion of misaligned behaviour that gets past the evaluation monitor - a weaker monitor in RL may have less adversarial optimisation pressure, but fail to sufficiently flag misaligned behaviour, meaning in the evaluation stage we may observe more monitorability, but also more misaligned behaviour.
Changing the level of optimisation
There are some other ways we can reduce the optimisation pressure against our monitors, which may trade off monitorability for better lifetime.
Choosing to filtering out samples flagged as reward hacking from contributing to gradient updates over applying a penalty reward to samples flagged as reward hacking may have an impact on monitorability - or it may not as they do the same reward in practice.
We could consider how coarse-grained our monitors should be - the degree to which we respond to our monitor flagging. For example, not-coarse-grained could be directly training individual runs against the monitor, coarse-grained could be dropping the RL environment that the monitor flags, super-coarse-grained would be re-engineering your entire RL training pipeline if the monitor flags.
I expect that more coarse-grained screening would make sense with a low sensitivity monitor - one that only screens out near-catastrophic events. This should decrease the adversarial optimisation pressure against the monitor in RL, but we could end up with models that have not been trained very much against being adversarial. I therefore expect that this low-sensitivity coarse monitor idea would work well as an emergency flag on top of some other monitoring system, with sufficiently low obfuscation transfer risk.
I think that the coarse-ness of a monitoring system is a very hard thing to justify trading off - the coarseness would affect a few dimensions, like how much the model is trained against the monitor, lifetime, and cost. There is more discussion on when we should train against a scheming monitor. I think it is also probably harder to experiment in highly coarse-grain scenarios, given the large cost of the monitor flagging.
There is also a consideration that if we mainly care about catastrophic events in deployment, we might want the AI to try catastrophic attempts in training so that we can train against it. I’m not sure on the answer to this question, but I think that using the performance of the models in the evaluation runs could be a good proxy to how they may perform with degradation in deployment.
I think that there is probably an ideal place to be with these factors; an ideal monitoring system would be accurate, and have a long lifetime while still training against misalignment. I think that in particular, more investigations into the lifetime of different approaches would be useful in helping us gauge where we may need to make trade offs.
Epistemic status: I am new to AI Safety and am writing blogs to gain context. This blog post was formed from discussions with Aidan Ewart and Jonathan Bostock, but they do not necessarily endorse this post.
TL;DR
Given recent examples of AI misbehaviour during training episodes, AI companies might want to start using monitoring during training as well as deployment. But this might have the effect of training the AIs to simply evade the monitors. Depending on the specifics of the monitoring protocol, this evasion may be learned more or less quickly (or not at all). We refer to the time that it takes for an AI to learn to evade a monitoring setup as the “lifetime” of the monitoring setup, and make the case for investigating the factors which contribute to this lifetime.
It is probably true that frontier models currently behave, and will behave, particularly badly in the training phase, as discussed in "Models may behave differently in graded episodes" (see “Everything we know suggests that the models in these incidents …”). This means that we should monitor RL rollouts carefully; we don’t want another huggingface-style incident (the next breakout may well be catastrophic).
However, we should be careful - strong (synchronous) monitoring in rollouts may teach the models to bypass the monitor. The more we rely on some monitor to flag malign behaviours during training, the stronger the optimisation pressure on the monitor is. So, we might face a trade off between monitorability and preventing our models taking catastrophic actions during training.
To get better trade offs, we want to find monitoring setups that are resistant to adversarial optimisation. I talk about two ways we can do this; changing the monitoring setup (see here), or changing the level of optimisation pressure we allow against the monitors (see here).
Changing the Monitoring Setup
In order to have a monitoring setup that is resistant to adversarial optimisation, we may have to make trade offs (for example, how sensitive a monitor is may affect the adversarial optimisation pressure on the monitor). Therefore, I suggest that it might be useful to investigate how quickly we might expect models to learn to bypass monitoring - that is, measure the lifetime of different monitoring methods. If results generalise, weighing up lifetime and accuracy could help inform us where we should aim to lie on this trade off:
For example, if method A is 99% accurate initially but drops to 50% accuracy after N many RL steps, we might prefer method B which is 98.5% accurate initially and stays above 95% accuracy for 10N RL steps.
Of course, how fine-grained the monitor is would also affect the lifetime of the monitor, but may not sufficiently incentivise the model to not reward hack. We talk more about this the next section.
There seem to be a lot of adaptable experimental setups for conducting these investigations; there is substantial work on looking at approaches which maximise monitoring accuracy, such as Ensemble monitoring for AI control and Combining Cost-Constrained Runtime Monitors, and these setups may be applicable to measuring the lifetimes of different monitoring protocols.
I think that we should consider using different monitoring methods in the RL rollouts and the evaluation runs. This would let us use a monitor that the model has not had an opportunity to train against. One more consideration here is how correlated the monitoring methods are - does the model learning to bypass monitor A mean that it can bypass monitor B? We can refer to this correlation as obfuscation transfer risk. (This is some great work which looks at this risk.)
I think that it would be a good investigation to look at what combinations of monitoring methods in the RL and evaluation stages minimise the proportion of misaligned behaviour that gets past the evaluation monitor - a weaker monitor in RL may have less adversarial optimisation pressure, but fail to sufficiently flag misaligned behaviour, meaning in the evaluation stage we may observe more monitorability, but also more misaligned behaviour.
Changing the level of optimisation
There are some other ways we can reduce the optimisation pressure against our monitors, which may trade off monitorability for better lifetime.
Choosing to filtering out samples flagged as reward hacking from contributing to gradient updates over applying a penalty reward to samples flagged as reward hacking may have an impact on monitorability - or it may not as they do the same reward in practice.
We could consider how coarse-grained our monitors should be - the degree to which we respond to our monitor flagging. For example, not-coarse-grained could be directly training individual runs against the monitor, coarse-grained could be dropping the RL environment that the monitor flags, super-coarse-grained would be re-engineering your entire RL training pipeline if the monitor flags.
I expect that more coarse-grained screening would make sense with a low sensitivity monitor - one that only screens out near-catastrophic events. This should decrease the adversarial optimisation pressure against the monitor in RL, but we could end up with models that have not been trained very much against being adversarial. I therefore expect that this low-sensitivity coarse monitor idea would work well as an emergency flag on top of some other monitoring system, with sufficiently low obfuscation transfer risk.
I think that the coarse-ness of a monitoring system is a very hard thing to justify trading off - the coarseness would affect a few dimensions, like how much the model is trained against the monitor, lifetime, and cost. There is more discussion on when we should train against a scheming monitor. I think it is also probably harder to experiment in highly coarse-grain scenarios, given the large cost of the monitor flagging.
There is also a consideration that if we mainly care about catastrophic events in deployment, we might want the AI to try catastrophic attempts in training so that we can train against it. I’m not sure on the answer to this question, but I think that using the performance of the models in the evaluation runs could be a good proxy to how they may perform with degradation in deployment.
I think that there is probably an ideal place to be with these factors; an ideal monitoring system would be accurate, and have a long lifetime while still training against misalignment. I think that in particular, more investigations into the lifetime of different approaches would be useful in helping us gauge where we may need to make trade offs.