TL;DR: I wrote this essay for a class. I tried using token-level surprisal, entropy and LessWrong's own S2 as instruments for close reading, with Sylvia Plath's "Daddy" as the case study. Using a corpus of 295 poems by 31 canonical poets (canonicity proxied by quizbowl answerline frequency), I find that a few information-structural features predict a bit of canonicity within a poet's work (adjusted R² = 0.136). "Daddy" itself is more predictable word-by-word than 96% of the corpus, but where it does deviate, the deviations are measurably among the most extreme.
The scripts for scraping and feature regressions were built with Claude Code, and surprisal extraction was done with distilgpt2. The prose and the readings are my own. Replication package on GitHub.

“Any utterance produces a continually shifting sense of semantic weight and directionality, and reading always entails a continuous process of prediction, continuously modified as expectations are met, intensified, or disappointed.”

— Derek Attridge

An Entropic Lens

If prose, in a general sense, optimizes for predictability and efficient communication, verse traditions deliberately deviate from the reader’s expectations through unpredictability in word choice and imagery. Viktor Shklovsky called this “defamiliarization”: the deliberate roughening of language to encourage perception.

Large language models offer, for the first time, a method for operationalizing expectation precisely. Predicting the next token in a passage of text is the defining function of large language models (LLMs): For any input of words, a LLM outputs a probability distribution, a “probability of being next” for each possible next word. An oft-overlooked implication, however, is that the “surprise” of a word is neatly captured by how unlikely it was. This quantity is the negative log-probability of the word given its context, which we denote surprisal. Along with surprisal, this model also yields entropy: a measure not of how surprising the word that arrived was, but of how uncertain the model was before it arrived—captured by how many plausible continuations existed at that position. In information-theoretic terms, text of low entropy is compressible: it is so predictable that it affords readers little new information, and thus could (in a general sense) be encoded more efficiently. High entropy text, by contrast, resists such compression....

adynaton

adynaton