if you can't mechinterp tiny, language-free model, you can't mechinterp big model (success! this bar has been passed)
if you can't prevent emergent scheming on tiny, language-free model, you can't prevent emergent scheming on big model
as above for generalization bounds
as above for regret bounds
as above for regret bounds on CEV in particular*
if you can do it on small model doesn't mean you can do it on big model. but a smoke test is that your method had better work on a model so small it can't talk. if your method requires a model big enough to talk in order to do anything that seems promising, you probably aren't robust against things that would arise even without the presence of language, and you might be getting punked by the "face".
* (which I call out because CEV is often treated as a thing where you need an enormous model to make any statement about it, which I think is importantly misleading, because I think if a CEV-correctness certification is ever going to be possible, it should be one that monotonically improves with scale, but produces only-finitely-bad bounds on small model)
What does it mean to prevent scheming on a tiny model? If we care about generalization performance rather than proving generalization bounds, to what distribution do we want the tiny model to generalize?
prevent emergent scheming: the true answer this minute is that I don't yet know, and would be using the "it should also work on a tiny model" as a guidepost to figure out how to answer this. I worry that any specific example I give won't be general enough, and I don't want to claim otherwise yet. doing that some now, though:
scheming might mean "reach reward region that is supposed to not happen but we can't easily de-reward-ify"?
if so, seems like maybe scheming is only happening if we don't get told the model is going to output actions that push it into... (read more)