This is a special post for quick takes by J Bostock. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I've been thinking about "trustedness" when it comes to AI, my current thinking goes something like this:
An AI is trustworthy on a given domain if we can build an eval which lets us predict the behaviour of that AI on that domain, when it is deployed.
For example, if you're doing OCR and your train, validation, test, and deployment datasets are all IID (like if you're reading in a bunch of old books from a set of a million in some library) then you can trust your OCR AI.
The problem is that IID is an insanely strong condition. This example rules out self-driving cars, since the training distribution will be data from before the car is released. Instead we might want to make a weaker claim.
Hand-wavey hypothesis: an AI is trustworthy if it is not using an internal abstraction which tracks whether it is being evaluated.
This made me think of Anthropic just straight-up zeroing the SAE features corresponding to evaluation awareness in Claude as part of their experiments.[1] I don't expect this to be a robust strategy as currently used[2], but maybe if you're building super robust causal models of LLMs, you can do something along these lines.