No LLM generated, assisted/co-written, or edited work.
Read full explanation
Epistemic status: a small, careful negative result from a time-boxed (~20h) project. One model, one benchmark suite, linear probes only, per-instance detection floor ≈ 0.04 balanced accuracy. Confident in the measurements; the interpretation is scoped accordingly.
Spent about 20 hours on a small question: if you override a robot policy's actions, does anything inside it register that the movement wasn't its own?
The setup is OpenVLA-OFT running LIBERO-Goal. It's memoryless — each call it just sees one camera image and the instruction, then emits an action chunk. So it has no way to remember what it asked for last time. Randomly, a quarter of the time I threw away the chunk it produced and executed a different one instead. Usually I substituted a chunk from another episode of the same task, so the robot still moved in a plausible way and my probes couldn't cheat by noticing weird motion.
Then I trained linear probes on the hidden states to see if a hijacked transition looked any different from a normal one. They found nothing. I tried handing the probe more and more help — the commanded chunk, the state from before the action as well as after — and it stayed at the floor.
The null is credible, because while the linear probes found nothing, the information is clearly available. If I hand-build features that compare what was commanded against what actually happened, a probe gets to 0.69, and up to 0.875 for the more disruptive overrides. The policy's intended action is also sitting right there in its hidden state, readable at R² = 0.64. So both halves of the comparison exist. Nothing in the model appears to have done the comparison.
What's more, the behavior is interesting. After a hijack the policy does get less confident on its next call, and that's a real effect. But it only shows up for the overrides that make the scene look unusual. The one where I zero out the motion entirely — the most blatant violation of what it asked for — produces no reaction at all. So it's reacting to what it sees, not to having been overridden. Ordinary visual feedback control is enough to explain everything here.
BTW, the most obvious limitation should be stressed. We only use linear probes, so the attribution information would be undetectable simply by hiding in the nonlinearity . A two-layer MLP on the same features would be the cheap way to show I'm wrong, and I'd like someone to try it. I'd also like to run this on an RL-trained version of the same model, since RL actually trains on the consequences of the policy's own actions.
Epistemic status: a small, careful negative result from a time-boxed (~20h) project. One model, one benchmark suite, linear probes only, per-instance detection floor ≈ 0.04 balanced accuracy. Confident in the measurements; the interpretation is scoped accordingly.
Spent about 20 hours on a small question: if you override a robot policy's actions, does anything inside it register that the movement wasn't its own?
The setup is OpenVLA-OFT running LIBERO-Goal. It's memoryless — each call it just sees one camera image and the instruction, then emits an action chunk. So it has no way to remember what it asked for last time. Randomly, a quarter of the time I threw away the chunk it produced and executed a different one instead. Usually I substituted a chunk from another episode of the same task, so the robot still moved in a plausible way and my probes couldn't cheat by noticing weird motion.
Then I trained linear probes on the hidden states to see if a hijacked transition looked any different from a normal one. They found nothing. I tried handing the probe more and more help — the commanded chunk, the state from before the action as well as after — and it stayed at the floor.
The null is credible, because while the linear probes found nothing, the information is clearly available. If I hand-build features that compare what was commanded against what actually happened, a probe gets to 0.69, and up to 0.875 for the more disruptive overrides. The policy's intended action is also sitting right there in its hidden state, readable at R² = 0.64. So both halves of the comparison exist. Nothing in the model appears to have done the comparison.
What's more, the behavior is interesting. After a hijack the policy does get less confident on its next call, and that's a real effect. But it only shows up for the overrides that make the scene look unusual. The one where I zero out the motion entirely — the most blatant violation of what it asked for — produces no reaction at all. So it's reacting to what it sees, not to having been overridden. Ordinary visual feedback control is enough to explain everything here.
BTW, the most obvious limitation should be stressed. We only use linear probes, so the attribution information would be undetectable simply by hiding in the nonlinearity . A two-layer MLP on the same features would be the cheap way to show I'm wrong, and I'd like someone to try it. I'd also like to run this on an RL-trained version of the same model, since RL actually trains on the consequences of the policy's own actions.