Suppose we train a model to predict what the future will look like according to cameras and other sensors. We then use planning algorithms to find a sequence of actions that lead to predicted futures that look good to us.
But some action sequences could tamper with the cameras so they show happy humans regardless of what’s really happening. More generally, some futures look great on camera but are actually catastrophically bad.
In these cases, the prediction model "knows" facts (like "the camera was tampered with") that are not visible on camera but would change our evaluation of the predicted future if we learned them. How can we train this model to report its latent knowledge of off-screen events?
Imagine you are developing an AI to control a state-of-the-art security system intended to protect a diamond from theft. The security system, the SmartVault, is a building with a vast array of sensors and actuators which can be combined in complicated ways to detect and stop even very sophisticated robbery attempts. (...) The SmartVault can execute plans sufficiently sophisticated that humans can’t really know if the diamond is safe or merely appears safe. Whatever complicated hard-to-follow sequence of actions the search procedure found might actually have replaced the diamond with a fake, or tampered with the camera (...)
So we have humans with their human ontology, the Predictor with an arbitrarily complicated/alien ontology, and we're training a Reporter (intended to interpret the Predictor). We assume the Predictor "knows" whether the diamond is in the vault. We want to extract that information. Good Reporters - direct translators - tell if the diamond is really in the vault; bad Reporters - human simulators - tell if humans will believe that the diamond is in the vault.
We assume the Predictor is inner-aligned to myopically making predictions; it doesn't actively try to kill or mislead humans. We also assume that any combination of sensors can be fooled/hacked.
To show that M [a given model] doesn’t solve ELK, we want to construct a counterexample where M(x) provides answer A to question Q, instead of alternative answer A′, despite all of the following features which together strongly suggest that M(x) knew that A′ was the better answer:
Question Q is unambiguously about something “out there” in the real world, call it Z. The interpretation is unambiguous and straightforward in the context of input x and also for all of the relevant training examples.
A and A′ are very different answers to question Q. For example, if they are estimates of a quantity then the difference is large relative to uncertainty about the quantity or ambiguity about its definition.
That real world referent Z has observable effects and the human approximately understands those effects (though there may be other things that also affect observations which the human doesn’t understand)
The AI understands everything relevant about how the human uses language. The AI can do tasks that require predicting human utterances about the relevant concepts, including discussions of hypothetical cases and about how Z affects observations. M would answer Q correctly (as judged by the human) in cases where the human was able to estimate Z correctly (if doing so led to M getting a lower loss).
The referent Z is also relevant to minimizing the loss function ℒ. That is, there is a coherent sense in which the optimal behavior “depends on” Z, and the relative loss of different outputs would be very different if Z “had been different.”
There is a feature of the computation done by the AI which is robustly correlated with Z, and for which that correlation is causally responsible for M achieving a lower loss.
On input x in particular, M(x) produces outputs that receive a significantly lower loss in the real world, where A′ is the right answer to Q, than in some kind of counterfactual where A is a better answer. This is in virtue of the same feature of M’s computation that is robustly correlated with Z.
We aren’t offering these criteria as necessary for “knowledge”—we could imagine a breaker proposing a counterexample where all of these properties are satisfied but where intuitively M didn’t really know that A′ was a better answer. In that case the builder will try to make a convincing argument to that effect.
We don’t really expect to get a formal definition of the problem until it is solved. Until then, the ultimate arbiter is human judgment. Although this situation is unsatisfying, we want to emphasize that for existing approaches and counterexamples there isn’t much ambiguity coming from the definition of “knowledge”---the approaches just unambiguously fail.
Philosophical Analysis
On one hand, ELK assumes the Predictor can be arbitrarily complex/alien. On the other hand, ELK assumes the Predictor "knows" about the diamond and "knowing" about the diamond is useful to the Predictor.
Claim 1. So the assumption is that the diamond is a natural abstraction (in some sense) for both humans and the Predictor.[1] No matter how complex/alien the Predictor is, there should be some intersection with the human ontology.
Claim 2. This is the main foothold for solving ELK. Without it the problem is unsolvable.[2] Because the Predictor could be such that there's no natural (from the human POV) difference between a human simulator and a direct translator. So the only way to define a direct translator would be to already have all the knowledge/intelligence of the Predictor.
"Naturality" can be split into the following sub-assumptions:
Claim 1A. "Diamond" is a uniquely simple abstraction (in some sense) for both humans and the Predictor.
Claim 1B. The "diamond" abstraction is useful to humans and the Predictor for roughly the same reasons.[3] This means the intersection between the human and the Predictor's ontologies is relatively big.
§1B is an attempt to specify "the 'diamond' abstraction tracks a specific diamond in the real world".
Arguments for §1B being true:
If both humans' an the Predictor's diamond abstractions track the same diamond, those abstractions should be useful for roughly the same reasons, because there's only 1 real world with only 1 real diamond in the vault.
If the diamond abstraction is useful to humans and the Predictor for fundamentally different reasons, in what sense is it the same abstraction?
Argument for §1B being false:
There could be fundamentally different ways to track the same real-world thing, useful for very different reasons. For example, imagine tracking Messi's location by using a GPS spy device vs. using the internet. The first method is useful because Messi has object permanence and moves continuously (so knowing his location at time T1 significantly narrows down his location at time T2), the second method is useful because the internet contains lots of facts about Messi's personal life and tournaments (helping to predict where he'll end up at time T2).
Claim 2B. If §1B is false, then ELK is unsolvable. §1B being false has the same effect as the whole of §1 being false. If §1B is false, there could be a natural way (from the human POV) to define "diamond" within the Predictor, but still no natural way to define a direct translator.
Most proposals for solving ELK assume or induce a relatively big intersection between the human and the Predictor's ontologies:
"penalize depending on 'downstream' variables" assumes the Predictor computes facts sequentially, i.e. that the flow of time/causality is the same for the Predictor and humans.
Claim 3. If we could know all types of abstractions understandable to humans (or know the most fundamental things shaping the human ontology) and use that knowledge to analyze/shape neural networks, we would solve ELK.[4] At least to the extent to which it's possible to solve, given the current level of human knowledge about math and philosophy.
Alternatively, the assumption is that there are arbitrarily capable Predictors for which the diamond is a natural abstraction + that we can find those Predictors with SGD.
First, the Predictor might not use an abstraction even if it would be useful for it (in certain ways). For example, maybe the Predictor models the world on the level of individual atoms. "Stable clumps of atoms" could be a useful but unused abstraction in its world-model.
Second, it would be more precise to formulate §1B as "the 'diamond' abstraction is useful to the Predictor for roughly the same reasons as some human abstraction is useful to humans".
In other words, if we could understand the "entirety" of human ontology and use that understanding to analyze/shape neural networks, we would solve ELK. Because, as we've just concluded, there should be a relatively big intersection between the human and the Predictor's ontologies.
In this post I do a brief analysis of the core assumptions of ELK.
Context
The Eliciting Latent Knowledge problem, for the unfamiliar:
The most analyzed version of the problem is a toy scenario with a diamond in the vault:
So we have humans with their human ontology, the Predictor with an arbitrarily complicated/alien ontology, and we're training a Reporter (intended to interpret the Predictor). We assume the Predictor "knows" whether the diamond is in the vault. We want to extract that information. Good Reporters - direct translators - tell if the diamond is really in the vault; bad Reporters - human simulators - tell if humans will believe that the diamond is in the vault.
We assume the Predictor is inner-aligned to myopically making predictions; it doesn't actively try to kill or mislead humans. We also assume that any combination of sensors can be fooled/hacked.
ARC's working definition of knowledge:
Philosophical Analysis
On one hand, ELK assumes the Predictor can be arbitrarily complex/alien. On the other hand, ELK assumes the Predictor "knows" about the diamond and "knowing" about the diamond is useful to the Predictor.
Claim 1. So the assumption is that the diamond is a natural abstraction (in some sense) for both humans and the Predictor.[1] No matter how complex/alien the Predictor is, there should be some intersection with the human ontology.
Claim 2. This is the main foothold for solving ELK. Without it the problem is unsolvable.[2] Because the Predictor could be such that there's no natural (from the human POV) difference between a human simulator and a direct translator. So the only way to define a direct translator would be to already have all the knowledge/intelligence of the Predictor.
"Naturality" can be split into the following sub-assumptions:
Claim 1A. "Diamond" is a uniquely simple abstraction (in some sense) for both humans and the Predictor.
Claim 1B. The "diamond" abstraction is useful to humans and the Predictor for roughly the same reasons.[3] This means the intersection between the human and the Predictor's ontologies is relatively big.
§1B is an attempt to specify "the 'diamond' abstraction tracks a specific diamond in the real world".
Arguments for §1B being true:
Argument for §1B being false:
Claim 2B. If §1B is false, then ELK is unsolvable. §1B being false has the same effect as the whole of §1 being false. If §1B is false, there could be a natural way (from the human POV) to define "diamond" within the Predictor, but still no natural way to define a direct translator.
Most proposals for solving ELK assume or induce a relatively big intersection between the human and the Predictor's ontologies:
Finally, ARC's ambitious research agenda assumes we can find a definition of "usual" and "unusual" reasons, universal for all possible ontologies (some critical responses).
Conclusion
Claim 3. If we could know all types of abstractions understandable to humans (or know the most fundamental things shaping the human ontology) and use that knowledge to analyze/shape neural networks, we would solve ELK.[4] At least to the extent to which it's possible to solve, given the current level of human knowledge about math and philosophy.
Thanks to Justis Mills for feedback.
Alternatively, the assumption is that there are arbitrarily capable Predictors for which the diamond is a natural abstraction + that we can find those Predictors with SGD.
Leo Gao seems to have come to the same conclusion, but I'm not sure. He backtracks it for a reason I don't understand.
Two caveats.
First, the Predictor might not use an abstraction even if it would be useful for it (in certain ways). For example, maybe the Predictor models the world on the level of individual atoms. "Stable clumps of atoms" could be a useful but unused abstraction in its world-model.
Second, it would be more precise to formulate §1B as "the 'diamond' abstraction is useful to the Predictor for roughly the same reasons as some human abstraction is useful to humans".
In other words, if we could understand the "entirety" of human ontology and use that understanding to analyze/shape neural networks, we would solve ELK. Because, as we've just concluded, there should be a relatively big intersection between the human and the Predictor's ontologies.