This is a special post for quick takes by Daniel Tan. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
It would be really great if we could somehow bottle the Janus eval taste for model character. I get the feeling there is real signal in how the cyborgism people approach model evaluations, etc, but it’s just totally unclear what to make of this stuff and what parts of it are real robust properties of LLMs vs just artifacts of the specific context / user / model