What does deception look like from the outside? I notice I am confused.
Imagine you have some algorithm which learns to predict sensory inputs, like a human. The internal structure of this will come to correspond to the external world in some way, as a part of generating a predictive model. Imagine the algorithm is walking around a building, looking at the objects in it. Most of the objects are ordinary, made-of-atoms objects. The algorithm learns to predict where the objects are, how they interact, and basically gets very low input-prediction error.
Then it comes across a teapot. The teapot isn't a real object, but in fact a hallucination projected onto the sensory arrays of the algorithm by a daemon. At first, the daemon only has to project some visual data. But then the algorithm picks up the teapot, so the daemon must project some tactile data.
For a "successful" deception, the algorithm must not "notice" that their purely-physical model of the world is being violated.
When the algorithm decides to make a cup of tea, the daemon must not only fake the sensations of a full, brewing pot of tea, but hide the sight of the water and teabag falling onto the floor. Later, the algorithm falls over by slipping on the spilled water, so the daemon must from then on fabricate all the sensory data that the algorithm gets.
Or, the daemon could fabricate some other reason for the sensory data. The water on the floor could have leaked in from a crack in the ceiling. For this to work, the algorithm must have already been uncertain about whether there was a crack in the ceiling or not, and the daemon must have known this.
This highlights two ways you can deceive a learning algorithm. Both involve control of all the information flowing between a system (in this case the whole world) and an algorithm. One requires modelling of the system, the other requires modelling the algorithm.
How else can we look at this? It seems like if there's a daemon making a fake teapot, the process of moving information has a discontinuity around the part of the algorithm defining a model of the teapot. The reason that a real teapot leads to a model of the teapot in the algorithm follows a different-looking causal chain to the reason a daemon leads to a model of the teapot in the algorithm.
This discontinuity defines a region which grows as the fake object interacts with the real ones. This can be expanded to fill the whole of the system external to the algorithm, or be contracted to zero. I hypothesize that these are the only two stable states of the region, every other boundary is broken by leaky abstractions.
The reason this feels like deception is that we also have a bit of self-knowledge which looks like . When we're deceived this is violated on a specific, local scale. Hence the discontinuity in .
Whether or not this discontinuity can be distinguished from other discontinuities in the function I'm not sure. At a first glance it seems different but I notice I am confused.
If these properties hold for everything humans would consider deception, this is important. One problem with translating ontologies (which is importantly relevant for the Eliciting Latent Knowledge problem) is how to identify deception. If for any learning algorithm we can specify a single world-plus-algorithm-state which has no deception, and specify which processes for changing that into other world-plus-algorithm preserve the no-deception property, then we can specify a "natural ontology" for that learning algorithm.
This reminds me of the proverb:
The more the algorithm probes the deception, the more effort the deceiver has to put into maintaining the illusion. Reality is completely self-consistent, so the accumulation of enough evidence into a sufficiently coherent world model will always resolve these types of deceptions if no further effort is applied to maintain them.
When I think of deception, though, I typically imagine a deceiver trying to create a strategic mismatch between their beliefs and another agent's beliefs. This information asymmetry is typically over something that the deceiver does not expect the agent to be able to investigate easily. It then allows them to get away with something that is in the deceiver's interest but against (what they think is probably) the other agent's interest.
It's usually something like a claim about the health benefits of snake oil (where the agent lacks the time or the resources to perform or look up a randomized controlled trial), rather than a claim about the existence of a physical object that's visible and within reach.
Maybe magic tricks go that route, though. Although those sorts of deceptions seem to be more about creating a superstimulus for the audience's curiosity drive, since the audience knows that their senses are being deceived and yet go out of their way to experience that deception.
You make some really excellent points here.
The teapot example is atypical of deception in humans, and was chosen to be simple and clear-cut. I think the web-of-lies effect is hampered in humans by a couple of things, both of which result from us only being approximations of Bayesian reasoners. One is the limits to our computation, we can't go and check a new update that "snake oil works" against all possible connections. Another part (which is also linked to computation limits) is that I suspect a small enough discrepancy gets rounded down to zero.
So... (read more)