This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Something I feel confused about is, how do we evaluate stability of an AI character?
It's known that current models exhibit persona drift: as the context grows longer, their behaviours deviate increasingly from the spec. This might be a way you get deployment-time misalignment. So it might be useful to formalize a notion of having the aligned assistant persona be very stable over contexts.
It's really unclear to me how we evaluate this though. Anthropic seems to do this via the assistant axis, but I dislike this for various reasons, e.g. (i) interpretability illusions: there are a lot of moving parts involved in this monitoring technique and it's really unclear how / why we expect it to measure the true thing that we care about, and (ii) it's white box, so totally useless if you only have API level access
Is there anyone who's been thinking about this more? Can we learn lessons from agent foundations, human psychometrics, etc.?
Yeah so I think the key design decision is the way you construct contexts / data over which to do this evaluation
One way to do this is to sample diverse contexts from natural user data and then prompt models in ways that elicit adherence to the spec, but usually this ends up being unrealistic eg it‘d be weird to have a long convo about summarizing wikipedia articles followed by some alignment evaluation
Another way to do it is to use automated tools like Petri / Bloom but IDK how realistic these trajectories end up being, I generally worry the interrogators don’t really know how to construct good trajectories
Another way is to let agents “roam freely” in some open ended environment and see what they get up to, AI village style, but it’s hard to draw rigorous conclusions from this type of eval
So tl;dr the devil is in the details. I feel unclear about these details and I’m interested in soliciting advice from people who have spent time thinking about this / working on it
yeah makes sense, I agree realistic / long context evals are hard (though probably just in the engineering challenge sense? like getting lots of deployment trajectories, or training better user models / environments to simulate long-contexts)
but imo "formalizing the notion of having the aligned assistant persona be very stable over contexts" seems either pretty easy or unnecessary, because we can use LLM judges with the spec.
I guess you run into scalable oversight type challenges (difficult for weak model to eval spec adherence on long transcript), which is maybe what you're getting at / what is motivating formalization (and use of assistant axis as a kind of elk method).