an off the cuff convo with jacob hilton about the arc white box metric. not expecting this raw convo to be super useful to people but lots of ideas here i expect to point to in the future. also shouldn't be construed as definitive reference on anything, this was a super chill convo

Leo Gao

anyone have strong takes on white box red teaming as a metric for (auto) interpretability? the claim is that you have a good interp method if it lets you construct surprising jailbreaks or otherwise surprising inputs. the main objection i can think of is perhaps approximately none of the interesting inputs are efficiently findable (consider a model that does a sha256 hash of the input and does something evil only if the first 128 bits are zeros). but surely even then it's a good metric of progress?
it would be nice to expand it to pseudoinputs but it's much harder to make an adversarially robust definition of pseudoinput
whereas it's pretty clear cut if you’ve made an input that makes the model tell you how to make a bomb
[maybe] you introduce a bottleneck where the majority of the compute is spent producing a latent z that a cheaper model can use to do a wide range of red teaming attacks given arbitrary targets
the hope is that z encodes an explanation of a new interpretability technique, or a new data structure that explains a lot of things about the model
the claim is this makes it harder to do boring attacks and you need to come up with something generalizable

do you have takes

Jacob Hilton  [4:19 PM]

you introduce a bottleneck where the majority of the compute is spent producing a latent z that a cheaper model can use to do a wide range of red teaming attacks given arbitrary targets

fwiw this is very similar to our research agenda, except that we require z to let you answer a wide range of estimation problems (and we call it an "explanation" rather than a "latent")
(we also want the estimation algorithm to be some simple thing we designed ourselves, rather than another model – all the complexity should live inside z)
We also want to be able to answer questions like conditional estimation problems, e.g. "conditional on the input looking realistic to some verifier, what's the probability the model defects?"
This is maybe the closest thing to your pseudoinputs
(And the original version of just finding inputs doesn't work for cryptographic reasons, as you say)

Leo Gao  [9:26 PM]
how do you do the conditional estimation thing?
also i assume the part where i use a relatively lower compute LM as the estimation algorithm is not a fatal decision, right? it's less elegant and gives less assurance but seems probably fine in practice
similarly, is the cryptographic objection actually fatal? it feels like the kind of thing where it might just be fine in practice

Leo Gao  [9:52 PM]
also how would you ideally pick the estimation problems? the reason i specify red teaming is because intuitively in some sense you want to find inputs that are impressive to find (and i guess in your setting, you want things where probability is very close to 0)
Jacob Hilton  [8:08 AM]

how do you do the conditional estimation thing?

It's going to depend on the structure that's present, but as a really dumb example if (X,Y) is multivariate Gaussian then you can compute conditional expectations like E[X|Y=y]. Or in a more general setting suppose you're trying to consider the conditional probability of an event P(A|B) = P(A and B)/P(B). If you understand the reasons why A and B might co-occur, you can probably do much better than separately computing P(A and B) and P(B) and dividing.

also i assume the part where i use a relatively lower compute LM as the estimation algorithm is not a fatal decision, right? it's less elegant and gives less assurance but seems probably fine in practice

I'd expect it to give less insight into how to solve the problem more generally, but on some level yes, if it works it works. LPE should be directly useful for mitigating tail risks so it's reasonable to just produce as good LPE estimates as possible.

similarly, is the cryptographic objection actually fatal? it feels like the kind of thing where it might just be fine in practice

I think this is again a question of research taste. If you are trying to aim for a principled, scalable solution you will probably be more troubled by cryptographic counterexamples than if you have a practical goal in mind and just want to optimize it.

also how would you ideally pick the estimation problems?

Hard to answer in generality - our approach is generally to try to tackle the "next hardest feasible case" so that we can make steady progress. E.g. trying to tackle cases with some new kind of structure we haven't been able to handle before. But there's definitely a worry that this is like trying to build a bridge across the ocean and I wouldn't want everyone to take this approach.
Leo Gao  [6:06 PM]
sure agree ultimately you it's a taste/judgement thing. I'm trying to basically take the ARC agenda and come up with an aggressively empirical version to do at oai. to me the LM as estimator feels very chill, but the cryptographic thing i feel divided on - it feels very likely true that even just in input space if you solved interp you would have so many more adversarial attacks then currently, and likely false that this would actually surface all of the scary behaviors
i kinda want to do jailbreaks because (a) it would probably be pretty cool if we solved jailbreaks, (b) people have already somewhat optimized for jailbreaks not existing so it's a nontrivial problem
Jacob Hilton  [6:27 PM]
I do think from a practical perspective, if you have good interp you should be better at coming up with adversarial examples/jailbreaks. There are old examples of this e.g. https://distill.pub/2021/multimodal-neurons/?utm_source=chatgpt.com#typographic-attacks
Leo Gao  [6:27 PM]
yep
Jacob Hilton  [6:29 PM]
wait sorry I'm thinking of the polysemantic neuron attack
Leo Gao  [6:31 PM]
yeah i know the steinhart paper you're referring to

========================================================================
(second thread, happening roughly the same time as the thread above)

Leo Gao  [2:53 PM]
also unrelatedly why does the white box challenge use random networks instead of trained networks? and would it be crazy for me to launch a parallel contest with small simplestories transformers

Jacob Hilton  [3:37 PM]
It's mainly because they're significantly harder, see some discussion here https://www.lesswrong.com/posts/Kben8CzS4awCwNw5c/announcing-the-arc-white-box-estimation-challenge?commentId=YWCaGNPvBXbvaXzdo
In general I am excited about pushing on the same goal (mechanistic estimates competitive with sampling) for small transformers, although I have soured on the contest format a bit
The difficulties are similar to trying to get LLMs to do work for you (which isn't particularly surprising, given most contestants seem to be using LLMs a lot)
Namely being very non-robust to slight misspecification of what you care about/not being able to handle fuzziness
It's a bit too early to say though: we haven't gone through final submissions for Phase 1 carefully yet, and we're changing the width/depth parameters for Phase 2 to something hopefully more reasonable. But my general impression is we're not super likely to see research insights we find exciting

Leo Gao  [3:47 PM]
hmm that's very sad - i was hoping this would make a good autoresearch target
aren't real models harder in some ways and easier in others?
like the existence of structure surely is very important

Jacob Hilton  [3:50 PM]

I was hoping this would make a good autoresearch target

I think it has potential, I just feel more optimistic about more human-in-the-loop versions right now, as opposed to just letting it rip

Leo Gao  [3:51 PM]
and low probability estimation is hard to cheat on, in the sense that if a lot of the action in real models is actually in the noise, then you would notice because you'd fail to make good progress on LPE

Jacob Hilton  [3:51 PM]
The existence of structure is what makes it harder, can't think of anything about that that makes it easier though

Leo Gao  [3:52 PM]
i mean as a concrete example if you open up the model and see a sha256 hash thing that only does x if the first n bits are whatever then this is clearly easier to do LPE on than a random model

Jacob Hilton  [3:52 PM]
Agree LPE is hard to cheat on, although I think if you just want to take incremental steps on LPE you're going to end up with a bunch of QMC/MCMC stuff that looks very unlike interp

Leo Gao  [3:52 PM]
and the claim would be that if anything practical models are more like this than like random models

Jacob Hilton  [3:53 PM]
Yes sorry I think very overtrained models on simple tasks could be easier than random models, you're right

Leo Gao  [3:53 PM]
what's the intuition for why this doesn't extend to reasonable normal models?

Jacob Hilton  [3:55 PM]
I'm not convinced interp has the tools needed to deal with "messy" trained models (even small ones trained on simple tasks, like AlgZoo)
I guess I don't have a strong reason a priori why these models aren't easier than random models in some ways. After all, they are very overtrained, so maybe there is less "noise" there in some sense. But they are just much harder.

Leo Gao  [4:01 PM]
is the idea something like, when you try to explain these models, you have some structure that explains maybe 80% of the variance, and then noise makes up the remaining 20%; and this is fine for coarse predictions but if you want to estimate low probability the noise dominates?

Jacob Hilton  [4:02 PM]
No it's messier than that
It's more that there isn't a clean distinction between the two
Like, it some sense there is no noise. There is a just a fixed set of weights
But when you analyze the weights, maybe you say, "well the weights are basically just this structure plus some noise"
And that modeling assumption gives you a way to make estimates.. e.g. you can say "this neuron is about 7 when the picture is a cat, sure it's sometimes a bit bigger, but that's just noise, it doesn't have any interesting interaction with anything"
But it was your choice as the interpreter to say that something was noise (and maybe you were wrong if you analyzed the model more carefully)
And importantly, you have to treat some stuff as noise in this kind of way, because if you analyze the model too carefully you take exponential time
So mostly the random network stuff is basically a way of asking: can we do the part of the analysis that just involves handling the things we want to treat as noise?
And for most trained models we think we still have to treat some stuff as noise, but we also have to handle the actual structure

Leo Gao  [5:57 PM]
hmm one thing that i’m confused about is, suppose you are correct that the stuff is just noise. then should this not cash out as your predictions about low probability events being correct? (and if you incorrectly missed some important structure, then you will heavily underestimate the chance of some rare event or something)

Jacob Hilton  [6:00 PM]
it should cash out as your predictions being more accurate as a function of the amount of compute you spent to compute the estimate, yes
but you won't be perfectly accurate, for the same reason that if you model sha256 as random and estimate the proportion of strings for which the first output bit is 1, you won't be perfectly accurate
sha256 is a kind of degenerate case where you can get almost perfect accuracy very cheaply (the answer is very close to 1/2), and then additional compute gets you extremely tiny benefit (you probably can't do much better than just checking inputs one by one)

Leo Gao  [6:06 PM]
i guess the question is, in practice is it good enough? like as long as it shows up in your LPE metric then it's good enough for auto interp / contest right?
sha256 seems uninteresting because we will basically never know (assuming no major cryptanalysis breakthroughs) whether it's even >1/2 or <1/2

Jacob Hilton  [6:10 PM]
our "matching sampling principle" conjecture is basically the claim that it is enough, i.e. the optimal estimator will be very interp-like and not just blindly sample
(caveats about the object you called "z" potentially being expensive to find)
(also agree sha256 is uninteresting, it's just an illustrative example)
it's a different question whether current LLMs are strong enough to find optimal estimators though