Eliciting Latent Knowledge problem for the unfamiliar:
Suppose we train a model to predict what the future will look like according to cameras and other sensors. We then use planning algorithms to find a sequence of actions that lead to predicted futures that look good to us.
But some action sequences could tamper with the cameras so they show happy humans regardless of what’s really happening. More generally, some futures look great on camera but are actually catastrophically bad.
In these cases, the prediction model "knows" facts (like "the camera was tampered with") that are not visible on camera but would change our evaluation of the predicted future if we learned them. How can we train this model to report its latent knowledge of off-screen events?
So we have humans with their human ontology, the Predictor (the model we need to interpret), the Planner (can be ignored for simplicity), and maybe we're training a Reporter (intended to interpret the Predictor).
We assume the Predictor is inner-aligned to myopically making predictions, it won't strategically mislead us by outputting wrong information at the right time. We also assume that something strongly correlated with the diamond (Z) is important for the Predictor's ability to make predictions (this is ARC's working definition of knowledge).
I was wondering whether the problem is possible to solve with brute force, at least if we have unlimited compute:
We do something like this with models of physics. For example, we prove that general relativity reduces to Newtonian mechanics under normal conditions. With enough compute I could take String Theory and check all the 10^500 possible universes to find ours. Since the Predictor is inner-aligned, it can be treated basically as a model of physics... (= this might require making an additional assumption, but ARC is ready to make it I think)
I assume there are fundamental obstacles to this, but what exactly are those obstacles? Maybe we can't define a universal language which would allow to prove the reduction of one model to another? Something else?
The most similar proposal to this I remember is "have AI help humans improve our understanding". It is dismissed for being uncompetitive. Yet uncompetitive ELK is arguably not actually solved and there are deeper reasons to reject the proposal. It relies on more roundabout methods than just brute forcing how one thing maps to another.
Now, maybe I'm just restating the problem... but right now I don't think so. I'm saying there should be a principled way to just prove how exactly one model reduces to the other instead of using arbitrary tricks to create a Reporter. If there are multiple, equally good ways to map human physics to the Predictor, then it would probably mean human physics aren't grounded in anything at all.
...
Maybe one of the core problems is that the Predictor might be a mess of high-level and low-level variables? So we'd first need to find a way to transform the Predictor into a low-level model of physics / establish the hierarchy of variables. How hard is this problem?