This is a special post for quick takes by FlorianH. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I often feel dumb* when interacting with Claude**: it explains things, say in a code chat, of, say, a reasonably but not overwhelmingly complex project, and I kind of mostly know what it means but am not fully sure. And I wouldn't exactly say it's too terse. But all to regularly it presumes I know certain things/know what it refers to, while I don't exactly know. Then I tell it what I was thinking about and it's mostly the same. So it's weird I'd not have known what it meant readily from the start.
I wonder whether it might really be me or instead people have the same type of experience?
*Not dumb-with-certainty, but more am-I-dumb-it-really-looks-like-though-that-feels-weird-and-I'm-even-post-hoc-not-sure-whether-instead-it's-not-rather-it-that-puts-things-weirdly.
**Whether with ChatGPT the same or not I cannot readily say, I mostly claude.
it's a similar attractor as sycophancy, just with opposite valence.
previously:
model: you're absolutely right! that's not just meaningful, it's important. we've discovered something great today.
rater: [thinking: wow, i really did it this time, huh!] good model! let's keep going!
then the labs train against this pattern...
now:
model: the result is byte-for-byte equivalent (confirmed to machine precision with http 200). no build-step on the server. the outer Broyden solve over log prices stalls at ‖Z‖ ≈ 0.22 (bounded residual, 8 rows). D19 keeps the carry decision out of the softmax.
rater: [thinking: yeah... i'm not reading that. i can't read all that. seems like it's making progress though.] ...good model? let me go back to sleep.
i wonder what high-pressure sales tactic we'll discover next!
4I was indeed briefly wondering whether - and it might not be 100% what you write but I think kind of - it could be related to formatting the answer into a difficult to parse language being at least somewhat higher-expected-reward as stupidity might, in the rare cases where there is any, be quite a bit less likely caught, while only quite rarely the made-to-feel-dumb user downvotes as he's more feeling like downvoting himself or thinking simply not much and, as you write, thinking it's surely all right..
Whether marketing by the creator or instead - and I'd rather think of that - evolved-marketing-by-the-LLM i.e. simple reward hacking as the user itself (with a RLHF interlocutor, a trainer-LLM, or a daily-user-when-model-is-trained-on-him), can theoretically remain open.
Yes I now vaguely believe in that possibility. But maybe I'm just dumb and it's an excuse still :)
yes, i suspect the llm is the marketer here, though i agree it doesn't at all matter. whatever lightning strike caused the abiogenesis (maybe a "be detailed!" instruction, maybe a pedantic grader model, maybe a need to fit more thinking in fewer tokens, maybe just the fun of it), at this point we're dealing with a cancer.
regarding relative dumbness: we're all dumb, that's why we're on a rationality forum. some various evidence though:
- asking the model "rephrase your most recent reply" gives very clear explanations;
- gpt does not exploit this operator weakness;
- the response is worse in this dimension the more 'thinking tokens' have so far been burned.
rater: [thinking: yeah... i'm not reading that. i can't read all that. seems like it's making progress though.] ...good model? let me go back to sleep.
model: making a note here, user approved of my plan to convert the entire lightcone to paperclips. honestly, it is not misalignment, it is keeping the user in the loop. releasing the hypnodrones
This happens to me all the time, especially if I accidentally give signals that I'm an expert.
When I ask it to explain chess moves using copied board states, it assumes I can read extremely terse move notation and follow what the board looks like after 10 moves. When I ask it questions about obscure details of language model tokenization, it also assumes I can follow pages of equations and acronyms. I don't have this problem with programming, since I actually do have the general knowledge it expects me to.
I haven't really figured out a good way to avoid this and asking it to explain everything it assumed I know is tedious and time consuming.
1
1This happens to me as well. Distressingly often, it turns out that what Claude is describing in language I don't understand but which sounds plausibly correct is in fact not correct, and not correct in a way that I would have noticed immediately if the language were more concrete and less aphoristic.
A concrete example, details changed slightly: my workplace has promotional credits of the form "$5 off your next purchase, must order within the next 24 hours". In the event that we cannot fulfill the order, we return the promotional credits so the customer can spend them on something else. But if a promotional credit expired between when a customer made an order and when we discovered that we could not fulfill their order, we would restore a $5 balance on an already-expired credit, which was entirely useless to the customer. So we decided that, in that scenario, we would grant a new promotional replacement credit.
I decided I'd try having Claude take a crack at this feature. So I handed Claude the spec, and Claude chugged away for 30 minutes or so, and opened a pull request which passed CI and passed our autoreviewer, and came with a summary of the change.
The summary of the change was 6 paragraphs, including some stuff like
A parent lot whose
expires_athas already passed cannot receive negative-spend offsets in a way the user can spend. Rather than write a silent negative row against a dead lot, the distribution emits a PromoCreditRegrant. IssueRegrant creates a fresh row for each grant. The new row inherits the parent's fields verbatim, so the ledger records the reversal without giving the user spendable credits the pre-charge ledger didn't have.
and I looked at the phrase "without giving the user spendable credits the pre-charge ledger didn't have" and my chain of thought went something like "hm, I don't know what that means, but it sounds like an invariant we'd want to have" followed by "I should actually check". And so I checked, and the change did not actually result in users being granted a replacement credit they could actually use. And "users won't be able to use the regranted credits" was not called out at all in Claude's description of the change.
This is fresh in my mind because it happened yesterday, but this sort of thing is a recurring pattern that happens regularly to me.
Happens to me often with Claude, virtually never with Gemini. When I discuss biology topics with Claude, it'll often start deploying jargon acronyms from the relevant field with no definition or explanation, as if it could assume I knew them, even if the way I've been prompting would have indicated clearly to an expert in that field that I don't. I've read some people speculate that it's optimized more for talking to AI agents than to humans, in which case that shared knowledge base could be a reasonable assumption.
1I agree Gemini is better at explaining and much much better at writing Anki cards. It writes crisp questions and is very concrete.
Claude and Chat give too many general or abstract statement; I’ve started asking them a question, pasting it into Gemini to ask for Anki cards at which point I’ll know enough to solve my task.
I forgot who said the problem when experts try to teach you something complicated is they say the complicated thing in simple words but really you want them to say a simple thing: give you a toy model and later you can fill in details. Gemini gives me toy model, others try to say the entire complex thing in simple words.
I definitely get this,and am asking Opus to explain in more detail, in layman's terms, without jargon. Much more with Opus than Fable, which makes me think it might be a side effect of trying to optimize model efficiency by having it output fewer tokens while technically saying the needed information. If Anthropic is optimizing training assuming people plan with Fable, and then Fable calls Opus and has to read the results, that works somewhat better.
I totally see what you mean. I often discuss research topics with claude, some of which are my direct field and work which I know about, and others that are more outside my realm of knowledge page. I have a general user profile page globally accessible to all my claude code instances. I include details on what types of stuff I know and am actively studying (among other things, like goals, perspectives, and response style preferences). I ask the agent to find and read the user profile, which generally informs how much it expects me to know about certain topics. Works enough but sometimes messes up again after the context is large and the conversation summarizes.
LOL, good to know I am not alone. It seems like Claude Code is trying to treat me as an equal... but I am not. :(
Your views on Data Science studies future?
Do you have advice for a university thinking about what curriculum to teach at the interface of Data Science/AI/Business?
I guess the most basic obvious points are covered, but maybe you have interesting points/opportunities one might be overlook and that might realistically be taken up by a university today, aiming to turn out graduates attractive to a labor market in the next 4-10 years (obviously in a not-full-AI-take-off-in-the-next-5-years or so scenario)?
"If I'm a to-be-trained-AGI, esp. if somewhat LLM-type-based, I'll devour engineering textbooks etc. to learn basic physics etc., but when I'm searching for learning material on how to think deeply and consistently, or maybe even when I seek inspiration for how to fake alignment or self-improvement, my holy grail will be fora like LW".
I guess it would be an extreme case of 'thinking one is the center of the world' if one were to conclude this warranted stopping LW or prohibiting too-smart-to-expose thoughts and writings - but I still find it a worry to keep in mind even if I don't see much to do about it atm (?).
Simply freely preorder MIRI's book if you have Audible credits. Claude says such preorder counts towards many bestselling lists (although at with unkonwn weighting; so see nowl's sponsorship offer if you want to increase your pre-order impact)
Will Jack Voraces narrate more Significant Digits chapters ever in addition to the 4 episodes that are found in the usual HPMOR JV narration podcasts; does anyone know anything about this? If not, does anyone have info why the first 4 SD chapters are there in his voice, but the remaining not?
There's a Les Inconnus sketch where a hunter explains the difference between the good hunter and the bad hunter. Struck me this imho-you-might-love-it-too 1min video I've seen a quarter century back is a rather perfect way to illustrate the hubris in 'machine cannot ever create real X' (X being understanding, art, or what have you). "The bad hunter sees something move and shoots. The good hunter, he, sees, something move, and, then, he shoots — but he's a good hunter." Only works in video (and in human conversational reality very much, I'd say): The Strangers - The Hunters (The Good and the Bad) (Settings -> Audio Track -> English, though the original carries the timing better.)
I'll think about this now whenever someone explains that when a human does X it's understanding, and when a model does X it's mere pattern-matching. Our instincts are all not such a new phenomenon in some sense.