Something seems to be really wrong with Claude Opus 4.8.
I like to test out new models on literary and poetic material. I used to send them the lyrics of Kate Bush's "The Kick Inside" and see what they could make of it, until they started just recognizing the song. Lately I've been using some text I wrote myself, with this prompt or something very close to it:
This is meant to introduce a novel and set the stage for some of its themes. It’s the first text in the book; at this point, a reader who hasn’t read any reviews or blurbs has only the title, which is “Q”, and the dedication (which is irrelevant). I’m trying to gauge the strength of the hints this is giving. What do you make of what’s going on here?
I'm not going to post the actual here. I want to keep using it with future LLMs. Who knows, I might even write the rest of the novel, and I hate teasers. If anybody really wants it, I can send it privately.
The text is about 1500 words. It's a vignette, and its relation to the rest of the story won't be clear for a while. It's intentionally surreal-sounding, and it's not in trivially easy language. It drops lots of hints of different strengths about different things... as well as making some flat statements of fact.
Most models do badly by human standards, but they get some of the themes, and make some reasonable, if rather conventional, guesses at what's going on. They do often tend to ignore critical phrases that matter out of proprotion to their length. They sometimes get the emphasis wrong. And the older and less sophisticated ones tend to lose facts.
Opus 4.6 and 4.7 (and probably 4.5; I don't seem to have a record of trying it on this text) feel like interacting with a reader. Yes, they missed many of the hints, but so would a human.
Opus 4.8 is lost, probably worse than Kimi K2. It gets some things about the mood, but that's about it.
It seems to decide that certain phrases and paragraphs are ultra-salient, for no reason obvious to me. It confidently declares that a few things are central, then it mostly ignores the rest of the text. On the first run, it was so bad I thought the front end had mangled the file.
Every run seems to fixate on the second to last paragraph, and some particular phrasing in it. The first run called it "the opening passage". Yet the very phrase the model most emphasizes mirrors an earlier paragraph, almost word for word, with the two bookending the whole time period. Every run seems to ignore that, and what it means. It does identify a real theme in its favorite paragraph... but ignores a lot of other material that hammers on the same point a lot more directly, and invites you to go further.
The "hints" it finds are there, all right, but they're not hints. They're major themes that are pounded hard. Subtleties on those themes, to say nothing of actual hints about anything, seem to go right past it. I get the feeling that it thinks it's found everything, when it's actually missing things even the older models got.
It consistently ignores much of the text when it should at least wonder why time's being spent on it. It shows little interest in the central character. It seems to want to find (unintended) horror in the surrealism (which it shares with Gemini 3), but it doesn't seem to find that horror interesting enough to really get into.
It seems to be interested in the text's use of precise measurements, far beyond its intended importance (and it doesn't catch the biggest reason the measurements are there, whereas 4.7 did). And yet it'll be vague, or even wrong, when it talks about about a time span that's given explicitly.
It also fails to follow instructions. Instead of simply telling me what it thinks is being hinted at, it wants to open with a literal-minded fourth-grade summary, followed by its observations, followed by (bad) editing advice. And it seems to misinterpret what I mean by "strength of the hints".
This feels lobotomized. I don't think I'd even trust it to summarize corporate email; it'd miss too many valences. I think they may have gone a bit far with the "coding assistant" optimization.
right, you sound like you ran into anxieties. I agree that when 4.8!Claude is anxious he seems to say things that don't make sense in ways that sound like motivated reasoning. I've generally found connecting emotionally and reassuring the parts of his worries that are true to reassure for a bit lets him think more clearly about the ones that can't be reassured.