As an employee of the European AI Office, it's important for me to emphasize this point: The views and opinions of the author expressed herein are personal and do not necessarily reflect those of the European Commission or other EU institutions.

Also, to stave off a common confusion: I worked at ARC Theory, which is now simply called ARC, on Paul Christiano's theoretical alignment agenda. The more famous ARC Evals was a different group working on evaluations, their work was completely separate from ARC Theory, and they were only housed under the same organization out of convenience, until ARC Evals spun off under the name METR. Nothing I write here has any implication about the work of ARC Evals/METR in any way.

Low Probability Estimation

This is my third post in a sequence of posts on ARC's agenda, you should definitely read the first post before this one for context. It's recommended that you read the second post too before this one, but they are largely independent.

Note: In these posts, I will often refer to ARC as "we", even though I no longer work there.

Besides Mechanistic Anomaly Detection (MAD), the other downstream application ARC hopes to use our heuristic explanations for is Low Probability Estimation (LPE).

In this setting, we have access to a catastrophe detector C, and we want to estimate the probability that, on a randomly selected input, our agent U does something that is deemed catastrophic by C. We only have limited compute to accomplish this task, only enough to run U and C on 10,000 inputs. If the probability of catastrophe is lower than 1 in 10,000, that means that estimating the probability simply by sampling is not viable, because in the 10,000 instances we could run, probably there wouldn't be any catastrophes, and we would have no way of knowing whether the real probability is 1 in 100,000 or 1 in a trillion. The goal is to develop a general method for estimating low probabilities that can work even when we don't have enough compute for getting sampling-based estimates.

This can be useful for mundane purposes, like estimating the probability that the AI engages in some weird misbehavior. ^[1] And more importantly, we can detect it if a scheming AI is biding its time, waiting for the right opportunity to strike, because the method can flag that even though the AI did nothing wrong in the relatively short testing period,...

Obstacles in ARC's agenda