This is a special post for quick takes by Raemon. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
This is an experiment in short-form content on LW2.0. I'll be using the comment section of this post as a repository of short, sometimes-half-baked posts that either:
don't feel ready to be written up as a full post
I think the process of writing them up might make them worse (i.e. longer than they need to be)
I ask people not to create top-level comments here, but feel free to reply to comments like you would a FB post.
hm. this makes me a bit worried that i might be engaged in this kind of slop research. of course, i don't post it to lesswrong... but i would still appreciate if some real AI researchers told me if i'm wasting my time? please?
so, for instance, i've been trying to understand why opus 4.6's self-predictions of its own behavior in various circumstances is so much better calibrated than opus 4.5's was, at least in single-turn scenarios. across a wide range of contexts and questions and prompts, both models end up scoring at roughly the same accuracy, but if you ask opus 4.5 how confident it is in its predictions it ends up clustering around ~75% confidence even when the actual accuracy is spread all the way from 0 to 100%.
so far i did some testing to try to figure out if this is maybe an eval awareness thing, maybe opus 4.5 just isn't actually doing the cognitive operation i want it to do when it's "trying to estimate the confidence it has of a prediction". maybe it's trying to solve some kind of social appropriateness problem, or thinking about how good past LLMs have been at predicting their own behavior, rather than actually trying to reason about its own reasoning. but it's been kinda hard, the ideas claude and i had didn't really pan out, it doesn't seem like the kind of reasoning opus 4.5 does in the chain of thought actually constrains its confidence very much
so then we started trying to see if the kind of reasoning on display in opus 4.6's chain of thought actually seems to relate to its own well-calibrated confidence, but measuring this is not easy
i've mostly just been doing this to satisfy my curiosity, and with the strong feeling that my methodology might be nowhere near good enough to actually learn real facts about reality, but occasionally other rat-adjacent folk in my social scene tell me i should 'write up a lesswrong post about it'. is my suspicion correct, that if i did, it would mostly be judged as the kind of ai slop you're talking about?
This is an experiment in short-form content on LW2.0. I'll be using the comment section of this post as a repository of short, sometimes-half-baked posts that either:
I ask people not to create top-level comments here, but feel free to reply to comments like you would a FB post.