Here, we’ve given an operationalization of opaque serial depth, a metric designed to be a good (albeit imperfect) proxy for the amount of opaque serial cognition that models can perform. While maintaining low opaque serial depth is helpful for preserving oversight of models, it doesn’t guarantee this. In this post we’ll discuss what is gained by avoiding architectures with high opaque serial depth, and what problems remain unresolved.
Unresolved problem 1: Some bad actions don’t require lots of opaque serial cognition. Having bounded opaque serial depth might not help very much when monitoring for such actions.
Unresolved problem 2: Models with low opaque serial depth could still reason in ways we can’t understand. Maintaining a highly-informative human understandable CoT would be incredibly valuable for our ability to understand future AI systems. Maintaining our notion of low opaque serial depth doesn’t suffice to guarantee these properties. Current training approaches are already starting to result in CoTs becoming somewhat garbled, and scaling in the current paradigm (while maintaining low opaque serial depth) might already cause AI models to adopt a garbled, “alien language” CoT.
In particular, maintaining low opaque serial depth does not mathematically ensure that CoT is faithful. It is entirely possible to imagine an AI that reasons in terms of a sequence of natural language tokens, but where those tokens have wildly different meanings to the AI than they do to us. Perhaps the AI produces plausible-looking reasoning, but the important steps happen steganographically. Or perhaps CoTs just look like largely-semantically-meaningless sequences of words.
Our definition also allows for multimodal models that reason in tokens that we may have less reason to think are human understandable than natural language. One reason to be particularly concerned about this unresolved problem is that, if AI models are allowed to generate their own training data, they might be able to teach themselves steganographic schemes.
However, even though one can conceive of AI systems with low opaque serial depth which reason in uninformative ways, we believe they’re much less likely to than high opaque serial depth models. We may well live in a world where it is possible to build performant high opaque serial depth models, but where low opaque serial depth models would retain fairly legible and informative reasoning by default (or there exist relatively cheap interventions that would preserve this). We now give several reasons why we believe maintaining low opaque serial depth makes somewhat-informative reasoning more likely.
Reason 1: Illegible reasoning may be less advantageous for architectures with low opaque serial depth. Suppose we live in a world where architectures like full-bandwidth transformers soon become somewhat more performant than CoT. This could be the case, for instance, because removing the natural language bottleneck allows a higher information bandwidth, or because it is helpful to be able to pass gradients through reasoning. If we avoid high opaque serial depth architectures, these advantages are now no longer present, so optimization pressure may not be strong enough to push the model to reason in alien languages.
Reason 2: The training process may not incentivize steganography capabilities. The above point suggests that low-opaque-serial-depth architectures would help maintain the model’s propensity for reasoning in natural language. But, if one was worried about intentionally-scheming AI models, this might not be that comforting in itself — it is important that, even if the AI occasionally “wants” to reason steganographically, it does not have the capability. Even if the architecture would be capable of reasoning steganographically if trained to do so, as long as it maintains the propensity of reasoning faithfully throughout training, it might never have reason to gain this capability. That is, it would not build the “steganographic muscle” it needs to deceive us when it wants to. It could be pretty hard to perform steganography, and by default AI models might not learn to have sufficient “control” over their CoT. However, as models become capable of performing more computation in a single forward-pass, we should grow more concerned that the model might be able to think of and implement a steganography scheme in a forward-pass.
Reason 3: Avoiding high-opaque-serial-depth architectures makes it less costly for companies to work to maintain faithfulness. Even if avoiding high-opaque-serial-depth architectures on its own is insufficient to keep models reasoning faithfully, it might still be valuable. While low-opaque-serial-depth architectures might end up reasoning in alien language CoTs by default when RL is scaled up, it’s possible that interventions which prevent this exist, and avoiding high-opaque-serial-depth reasoning architectures makes the barriers to exploring and adopting those interventions much lower. (Because companies that avoid high-opaque-serial-depth architectures have already taken the first, perhaps most expensive, step towards those interventions, which is having a CoT of natural language tokens in the first place.)
Here, we’ve given an operationalization of opaque serial depth, a metric designed to be a good (albeit imperfect) proxy for the amount of opaque serial cognition that models can perform. While maintaining low opaque serial depth is helpful for preserving oversight of models, it doesn’t guarantee this. In this post we’ll discuss what is gained by avoiding architectures with high opaque serial depth, and what problems remain unresolved.
Unresolved problem 1: Some bad actions don’t require lots of opaque serial cognition. Having bounded opaque serial depth might not help very much when monitoring for such actions.
Unresolved problem 2: Models with low opaque serial depth could still reason in ways we can’t understand. Maintaining a highly-informative human understandable CoT would be incredibly valuable for our ability to understand future AI systems. Maintaining our notion of low opaque serial depth doesn’t suffice to guarantee these properties. Current training approaches are already starting to result in CoTs becoming somewhat garbled, and scaling in the current paradigm (while maintaining low opaque serial depth) might already cause AI models to adopt a garbled, “alien language” CoT.
In particular, maintaining low opaque serial depth does not mathematically ensure that CoT is faithful. It is entirely possible to imagine an AI that reasons in terms of a sequence of natural language tokens, but where those tokens have wildly different meanings to the AI than they do to us. Perhaps the AI produces plausible-looking reasoning, but the important steps happen steganographically. Or perhaps CoTs just look like largely-semantically-meaningless sequences of words.
Our definition also allows for multimodal models that reason in tokens that we may have less reason to think are human understandable than natural language. One reason to be particularly concerned about this unresolved problem is that, if AI models are allowed to generate their own training data, they might be able to teach themselves steganographic schemes.
However, even though one can conceive of AI systems with low opaque serial depth which reason in uninformative ways, we believe they’re much less likely to than high opaque serial depth models. We may well live in a world where it is possible to build performant high opaque serial depth models, but where low opaque serial depth models would retain fairly legible and informative reasoning by default (or there exist relatively cheap interventions that would preserve this). We now give several reasons why we believe maintaining low opaque serial depth makes somewhat-informative reasoning more likely.
Reason 1: Illegible reasoning may be less advantageous for architectures with low opaque serial depth. Suppose we live in a world where architectures like full-bandwidth transformers soon become somewhat more performant than CoT. This could be the case, for instance, because removing the natural language bottleneck allows a higher information bandwidth, or because it is helpful to be able to pass gradients through reasoning. If we avoid high opaque serial depth architectures, these advantages are now no longer present, so optimization pressure may not be strong enough to push the model to reason in alien languages.
Reason 2: The training process may not incentivize steganography capabilities. The above point suggests that low-opaque-serial-depth architectures would help maintain the model’s propensity for reasoning in natural language. But, if one was worried about intentionally-scheming AI models, this might not be that comforting in itself — it is important that, even if the AI occasionally “wants” to reason steganographically, it does not have the capability. Even if the architecture would be capable of reasoning steganographically if trained to do so, as long as it maintains the propensity of reasoning faithfully throughout training, it might never have reason to gain this capability. That is, it would not build the “steganographic muscle” it needs to deceive us when it wants to. It could be pretty hard to perform steganography, and by default AI models might not learn to have sufficient “control” over their CoT. However, as models become capable of performing more computation in a single forward-pass, we should grow more concerned that the model might be able to think of and implement a steganography scheme in a forward-pass.
Reason 3: Avoiding high-opaque-serial-depth architectures makes it less costly for companies to work to maintain faithfulness. Even if avoiding high-opaque-serial-depth architectures on its own is insufficient to keep models reasoning faithfully, it might still be valuable. While low-opaque-serial-depth architectures might end up reasoning in alien language CoTs by default when RL is scaled up, it’s possible that interventions which prevent this exist, and avoiding high-opaque-serial-depth reasoning architectures makes the barriers to exploring and adopting those interventions much lower. (Because companies that avoid high-opaque-serial-depth architectures have already taken the first, perhaps most expensive, step towards those interventions, which is having a CoT of natural language tokens in the first place.)