This is a special post for quick takes by Anthony DiGiovanni. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I continue to be puzzled as to why many people on LW are very confident in the "algorithmic ontology" about decision theory:
So I see all axes except the "algorithm" axis as "live debates" -- basically anyone who has thought about it very much seems to agree that you control "the policy of agents who sufficiently resemble you" (rather than something more myopic like "your individual action")
Can someone point to resources that clearly argue for this position? (I don't think that, e.g., the intuition that you ought to cooperate with your exact copy in a Prisoner's Dilemma — much as I share it — is an argument for this ontology. You could endorse the physicalist ontology + EDT, for example.)
I can't point you to existing resources, but from my perspective, I assumed an algorithmic ontology because it seemed like the only way to make decision theory well defined (at least potentially, after solving various open problems). That is, for an AI that knows its own source code S, you could potentially define the "consequences of me doing X" as the logical consequences of the logical statement "S outputs X". Whereas I'm not sure how this could even potentially be defined under a physicalist ontology, since it seems impossible for even an ASI to know the exact details of itself as a physical system.
This does lead to the problem that I don't know how to apply LDT to humans (who do not know their own source code), which does make me somewhat suspicious that the algorithmic ontology might be a wrong approach (although physicalist ontology doesn't seem to help). I mentioned this as problem #6 in UDT shows that decision theory is more puzzling than ever.
ETA: I was (and still am) also under strong influence of Tegmark's Mathematical universe hypothesis. What's your view on it?
Thanks, that's helpful!
I am indeed interested in decision theory that applies to agents other than AIs that know their own source code. Though I'm not sure why it's a problem for the physicalist ontology that the agent doesn't know the exact details of itself — seems plausible to me that "decisions" might just be a vague concept, which we still want to be able to reason about under bounded rationality. E.g. under physicalist EDT, what I ask myself when I consider a decision to do X is, "What consequences do I expect conditional on my brain-state going through the process that I call 'deciding to do X' [and conditional on all the other relevant info I know including my own reasoning about this decision, per the Tickle Defense]?" But I might miss your point.
Re: mathematical universe hypothesis: I'm pretty unconvinced, though I at least see the prima facie motivation (IIUC: we want an explanation for why the universe we find ourselves in has the dynamical laws and initial conditions it does, rather than some others). Not an expert here, this is just based on some limited exploration of the topic. My main objections:
Not sure what you mean by "the math" exactly. I've heard people cite the algorithmic ontology as a motivation for, e.g., logical updatelessness, or for updateless decision theory generally. In the case of logical updatelessness, I think (low confidence!) the idea is that if you don't see yourself as this physical object that exists in "the real world," but rather see yourself as an algorithm instantiated in a bunch of possible worlds, then it might be sensible to follow a policy that doesn't update on e.g. the first digit of pi being odd.
query rephrase: taboo both "algorithmic ontology" and "physicalist ontology". describe how each of them constructs math to describe things in the world, and how that math differs. That is, if you're saying you have an ontology, presumably this means you have some math and some words describing how the math relates to reality. I'm interested in a comparison of that math and those words; so far you're saying things about a thing I don't really understand as being separate from physicalism. Why can't you just see yourself as multiple physical objects and still have a physicalist ontology? what makes these things different in some, any, math, as opposed to only being a difference in how the math connects to reality?
I think I just don't understand / probably disagree with the premise of your question, sorry. I'm taking as given whatever distinction between these two ontologies is noted in the post I linked. These don't need to be mathematically precise in order to be useful concepts.
ah my bad, my attention missed the link! that does in fact answer my whole question, and if I hadn't missed it I'd have had nothing to ask :)
There isn't a difference, though it's only because algorithms can simulate all of the physical stuff that's in the physical ontology, and thus the physical ontology is a special case of the algorithmic ontology.
Helpful link here to build intuition:
http://www.amirrorclear.net/academic/ideas/simulation/index.html
That's what I already believed, but OP seems to disagree, so I'm trying to understand what they mean
The argument for this is spelled out in Eliezer and Nate's Functional Decision Theory: A New Theory of Instrumental Rationality. See also the LessWrong wiki tag page
Thanks — do you have a specific section of the paper in mind? Is the idea that this ontology is motivated by "finding a decision theory that recommends verdicts in such and such decision problems that we find pre-theoretically intuitive"?
That sounds like a good description of my understanding, but I'd also say the pre-theoretic intuitions are real damn convincing!
There's a table of contents which you can use to read relevant sections of the paper. You know your cruxes better than I do.
shrug — I guess it's not worth rehashing pretty old-on-LW decision theory disagreements, but: (1) I just don't find the pre-theoretic verdicts in that paper nearly as obvious as the authors do, since these problems are so out-of-distribution. Decision theory is hard. Also, some interpretations of logical decision theories give the pre-theoretically "wrong" verdict on "betting on the past." (2) I pre-theoretically find the kind of logical updatelessness that some folks claim follows from the algorithmic ontology pretty bizarre. (3) On its face it seems more plausible to me that algorithms just aren’t ontologically basic, they’re abstractions we use to represent (physical) input-output processes.
From my perspective, I tend to think that the algorithmic ontology is better for decision theory in general because it's a strictly more general ontology than physicalism, with the physicalist ontology, including all of it's objects and models existing in the algorithmic ontology as a special case of the more general algorithmic ontology, so from a general perspective, it's better to work in the ontology that works in more cases than a physicalist ontology, while remembering that the physicalist ontology can exist and be valid as a special case of the algorithmic ontology.
I don't understand. It seems that when people appeal to the algorithmic ontology to motivate interesting decision-theoretic claims — like, say, "you should choose to one-box in Transparent Newcomb" — they're not just taking a more general perspective. They're making a substantive claim that it's sensible to regard yourself as an algorithm, over and above your particular instantiation in concrete reality.
Another way to say it is that the substantive part of the claim around decision theory that favors the algorithmic ontology is that your identity is closer to an isomorphism class/equivalence class of programs than a concrete instantiation of a program, akin to generic programming:
https://en.wikipedia.org/wiki/Generic_programming
https://en.wikipedia.org/wiki/Parametric_polymorphism
An argument for that is in an infinite universe where our patch of the affectable universe isn't special at the large scale in how it clumps up atoms into bigger structures, which is very probable then if you had arbitrarily good faster-than-light travel, you'd eventually meet copies of yourself, since there's only a finite number of possibilities for your identity, due to finite memory (conditional on you existing), but while it's combinatorially large, it's not infinite, and thus some states have to be repeated in an infinite universe, so your identity isn't unique under the instance-focused worldview (because there are infinite instances, not 1 instance), but from the algorithmic perspective they are one you, and thus you can trade with yourself.
The more practical argument is that a lot of the algorithmic ontology stuff was focused on future people who are closer to AIs today (but better), in that they can merge/copy to the extent that they have compute, and importantly are way more parallel in the sense that they can meaningfully have multiple experiences like current AI probably does today, and thus a lot of the issue with translating it to the human case is that humans are way more serial than AI.
you'd eventually meet copies of yourself
But a copy of me =/= me. I don't see how you establish this equivalence without assuming the algorithmic ontology in the first place.
Okay, my point here was that the copies would not only have the same algorithm, but also the same physical structure arbitrarily finely, and I don't need to assume the algorithmic ontology, I only need to remember that there's only a finite amount of configurations of atoms that end up in human bodies, meaning that the number of distinct identities is upper bounded by a finite number. The search space is combinatorically large, but not infinitely large, which ensures that in an infinite universe, some states must be repeated exactly.
That's why you'd meet yourself, eventually, it's not because of the algorithmic ontology, but because there isn't an infinite number of possibilities for your identity.
the copies would not only have the same algorithm, but also the same physical structure arbitrarily finely
I understand, I'm just rejecting the premise that "same physical structure" implies identity to me. (Perhaps confusingly, despite the fact that I'm defending the "physicalist ontology" in the context of this thread (in contrast to algorithmic ontology), I reject physicalism in the metaphysics sense.)
This also seems tangential, though, because the substantive appeals to the algorithmic ontology that get made in the decision theory context aren't about physically instantiated copies. They're about non-physically-instantiated copies of your algorithm. I unfortunately don't know of a reference for this off the top of my head, but it has come up in some personal communications FWIW.
Summary of why I don't buy diachronic Dutch book arguments
(Using Elga's Dutch book against imprecise credences as an example, where the agent faces a sequence of two bets A and B.)
(See also the links in the table here re: money pump arguments for completeness, which have a very similar structure.)
ETA (Jul 15, 2026): See here for a summary of my take on the objection: "But following C is behaviorally the same as rejecting imprecise credences (i.e., the imprecise credences don't do any work)."
H/t Jesse Clifton for making this salient to me; not sure if he'd endorse this version of the counterargument though. ↩︎
Indeed, Elga gives some great intuition pumps for this in the intro of his paper! ↩︎
You might say, adopting a different standard of rationality (as Elga asks the impreciser to do) is more psychologically tractable than C. But one way to achieve C is to adopt the principle of resolute choice. ↩︎
Hi Anthony. Here is a summary of Adam Elga's argument. Below is an outline of Adam's argument against the plan strategy. Which part of it do you reject?
2. PLAN — When you act, you simultaneously form a plan binding your later choices to cohere with it (reject Bet A → plan to accept Bet B → follow through), but without changing any beliefs. Elga refutes this with the case of Sally, who cares only about money and has a highly unsharp credence about rain. Compare two scenarios: in the first she rejected Bet A and is now offered Bet B; in the second she's offered Bet B alone. PLAN permits rejecting Bet B in the second but not the first. Yet the monetary consequences of accepting and of rejecting Bet B are identical across the two scenarios, and her beliefs are identical, and money is all she cares about—so the situations are alike in every respect she cares about. Rationality can't impose different requirements on choices that are identical in all relevant respects. To the rejoinder "but rejecting Bet B would break her plan," Elga replies that either plan-breaking is something Sally finds costly (contradicting the stipulation that it's costless for her), or it isn't—in which case "Don't break plans!" is as groundless a constraint as "Don't break mirrors!" He flags but sets aside the resolute-choice tradition (Gauthier, McClennen) that would defend plan-following.
I'm proposing binding commitments, not plans. As I say, there's no choice to be made after committing to C and rejecting bet A. So I reject the claim that the commitment response requires "imposing different requirements on choices that are identical in all relevant respects".
I see. Below is how Claude thinks Adam would object. Any reactions?
["corrected" because Claude initially said something that did not make sense. I pointed this out, and asked Claude to update what would be Adam's objections. The updated objections below make sense to me.]
1. The commitment doesn't pay the original bill; it relocates it. Elga's challenge was: how do unsharp credences themselves constrain rational action? Notice what C concedes. At the B-node the agent's credences-plus-maximality still say rejecting B is permissible — that is why she has to lock the option out in advance. So the imprecise state still delivers the wrong local verdict; C works by overriding it. What constrains action, then, isn't the unsharp credence at all — it's a self-binding policy bolted on top, one that mimics a determinate dominance-avoiding disposition. So the credence is idle with respect to the very choice that saves the agent, and indeed must be countermanded. That is confirmation of Elga's thesis (no acceptable account of how unsharp credences constrain action), not a refutation of it.
2. Required or merely permitted? For C to rescue UNSHARP, reject-both has to become impermissible. But is enacting C rationally required? If it's only permitted, then not-committing is also permitted, and DiGiovanni's own horn 2 says that a non-committing agent who then rejects both has done nothing locally irrational — so the manifestly-bad outcome is still reachable by a permissible route, and UNSHARP still licenses it. So C must be required. But then the requirement has to come from somewhere, and it can't come from the local maximality verdicts (which permit not-committing and permit rejecting B). It comes from a dominance-avoidance-over-policies norm — i.e., from evaluating the sequence ex ante. That is just the policy-level / SEQUENCE evaluation. C's distinctive contribution is only that, by physically removing the later choice, it dodges the identical-choices objection that Elga levelled at policy-level evaluation. So the commitment isn't an independent third option; it's ex-ante maximality plus a lock-out whose sole job is to keep Sally from getting a grip.
3. And that lock-out is itself the concession. Here is the line that doesn't depend on any of my earlier missteps. Elga's target is UNSHARP: it is consistent with perfect rationality to have unsharp credences. DiGiovanni's defense requires that the perfectly rational unsharp agent be a resolute self-binder who must delete options from her own future choice set to avoid a foreseeable dominated outcome. But an agent who has to handcuff herself against what her own beliefs-plus-decision-rule count as permissible is displaying a belief state so defective that it needs handcuffs — and a sharp agent needs none. The need for C is a symptom of the defect, not a vindication of its compatibility with perfect rationality. DiGiovanni's footnote reply — that adopting sharp credences is merely "more psychologically tractable," and resolute choice achieves the same thing — misses this: the point isn't tractability, it's that the resolute imprecise agent's good behavior is entirely explained by the commitment (which behaves like a determinate disposition), while the imprecision contributes nothing positive a sharp state couldn't and must be overridden at the crux. A state that must be overridden to avoid disaster, and adds nothing a sharp state lacks, has no claim to being what ideal rationality delivers.
4. The "you're just coercing the future self too" charge fails on a disanalogy. DiGiovanni's sharpest jab is that Elga's own prescription — "adopt sharp credences" — is itself a way of making your future self's beliefs deviate from what they'd count as locally rational, so it's no more legitimate than demanding C. But there's an asymmetry. "Adopt sharp credences" is not an instruction to the B-node self to act against its own current verdict; it's a claim about which belief state to occupy. The sharp agent at the B-node does not override anything — her local verdict, given her precise p, already tells her what to do, and it already coheres with her A-node verdict, automatically, with no binding. The imprecise agent's trouble is that her local verdicts are not automatically time-coherent, so she must either coerce herself (commit) or behave incoherently across time. Elga's SHARP diagnoses the fault as lying in a belief state whose local verdicts fail to cohere, and the fix is precision, not coercion. So "adopt sharp credences" means "have a state that doesn't need forcing" — which is the opposite of forcing.
Once the strict-rules horn is off the table, Elga doesn't have a clean knockdown of the committed imprecise agent. What he has is (2) and (3): the demand that C be required rather than permitted, and the charge that the very need to self-bind is a mark against perfect rationality. DiGiovanni can keep resisting by insisting that "imprecise credences + resolute choice" is simply a coherent rival package to "sharp credences," and that the Dutch book indicts only the third combination — imprecision plus inability to commit plus act-by-act choice. At that point you're back at the bedrock intuition Elga admits he can only pump: whether a fully-informed, money-only agent who ends up foreseeably dominated has thereby shown a rational defect, or has merely done something a differently-organized agent could have avoided. This is exactly the resolute/sophisticated/myopic fork from the money-pump literature you already know from Gustafsson — and DiGiovanni's own note that his argument "has a very similar structure" to money-pump arguments is him planting his flag on the resolute side.
So the corrected verdict is narrower than what I gave before: the commitment move genuinely escapes both the Sally objection and the strict-rules horn, and Elga's live reply is not "that collapses into sharp betting" (it doesn't) but "requiring self-binding to avoid dominance is itself incompatible with the perfect rationality UNSHARP asserts — and the credence is doing no work in the rescue anyway."
(I generally find LLM-written philosophy critiques overstate various things, which I don't think are worth the time to engage with. Just briefly replying here to the substantive points.)
"The imprecise credence itself doesn't guide the action w.r.t. the second bet" does not imply "you shouldn't have imprecise credences in general". Elga's argument doesn't tell us at all that we should, say, "go with our best guess" about altruistic interventions.
One's best guess for the intervention with the highest expected marginal cost-effectiveness (EMCE) may be wrong. However, one should still support it under precise credences? Greater uncertainty about which intervention has the highest EMCE will tend to make interventions decreasing that uncertainty rank higher.
one should still support it under precise credences?
I'm saying that Elga's argument doesn't tell us to have precise credences in the first place. It only tells us "you should commit to act in a way that avoids sure losses".
That makes sense. However, if one cannot make such a commitment, or finds its implications undesirable, Elsa's argument should update one away from unsharp credences (even if one ends up preferring these all things considered)?
Examples of awareness growth vs. logical updates
(Thanks to Lukas Finnveden for discussion that prompted these examples, and for authoring examples #3-#6 verbatim.)
A key concept in the theory of open-minded updatelessness (OMU) is "awareness growth", i.e., conceiving of hypotheses you hadn't considered before. It's helpful to gesture at "discovering crucial considerations" as examples of awareness growth. But not all CC discoveries are awareness growth. And we might think we don't need this OMU idea if awareness growth is just logical updating, i.e. you already had nonzero credence in some hypothesis, but you changed this credence purely by thinking more. What's the difference? Here are some examples.
Lesser-known LLMisms, in my experience:
Isn't the "you get what you measure" problem a problem for capabilities progress too, not just alignment? I.e.: Some tasks are sufficiently complex (hence hard to evaluate) and lacking in unambiguous ground-truth feedback that, when you turn the ML crank on them, you're not necessarily going to select for actually doing the task well. You'll select for "appearing to do the task well," and it's open question how well this correlates with actually doing the task well. ("Doing the task" here can include something much higher-level, like "being 'generally intelligent'.")
Which isn't to say this problem wouldn't bite especially hard for alignment. Alignment seems harder to verify than lots of things. But this is one reason I'm not fully sold that once you get human-level AI, capabilities progress will get faster.
(I'm hardly an expert on this, so might well have missed existing discourse on & answers to this question.)
Linkpost: "Against dynamic consistency: Why not time-slice rationality?"
This got too long for a "quick take," but also isn't polished enough for a top-level post. So onto my blog it goes.
I’ve been skeptical for a while of updateless decision theory, diachronic Dutch books, and dynamic consistency as a rational requirement. I think Hedden's (2015) notion of time-slice rationality nicely grounds the cluster of intuitions behind this skepticism.
Endorsement on reflection is not straightforward, even states of knowledge or representations of values or ways of interpreting them can fail to be endorsed. It's not good from my perspective for someone else to lose themselves and start acting in my interests. But it is good for them to find themselves if they are confused about what they should endorse on reflection.
From 0-my perspective, it’s good for 1-me to believe updatelessness is rational, even if from 1-my perspective it isn’t.
Values can say things about how agents think, about the reasons behind outcomes, not just the outcomes themselves. An object level moral point that gestures at the issue is to say that it's not actually good when a person gets confused or manipulated and starts working towards an outcome that I prefer, that is I don't prefer an outcome when it's bundled with a world that produced it in this way, even if I would prefer the outcome when considered on its own. So I disagree with the claim that, assuming "from 1-my perspective it's not good to do X", then it's still "from 0-my perspective it's good for 1-me to believe that they should do X".
A metaethical point on interpretation of states of knowledge or values not being straightforward is about the nature of possible confusion about what an agent might value. There is a setting where decision theory is sorted out, and values are specified explicitly, so that the notion of them being confused is not under consideration. But if we do entertain the possibility of confusion, that the design isn't yet settled, or that there is no reflective stability, then the thing that's currently written down as "values" and determines immediate actions has little claim to be actual values.
Claims about counterfactual value of interventions given AI assistance should be consistent
A common claim I hear about research on s-risks is that it’s much less counterfactual than alignment research, because if alignment goes well we can just delegate it to aligned AIs (and if it doesn’t, there’s little hope of shaping the future anyway).
I think there are several flaws with this argument that require more object-level context (see this post).[1] But at a high level, this consideration—that research/engineering can be delegated to AIs that pose little-to-no risk of takeover—should also make us discount the counterfactual value of alignment research/engineering. The main plan of OpenAI’s alignment team, and part of Anthropic’s plan and those of several thought leaders in alignment, is to delegate alignment work (arguably the hardest parts thereof)[2] to AIs.
It’s plausible (and apparently a reasonably common view among alignment researchers) that:
It seems that if these claims hold, lots of alignment work would be made obsolete by AIs, not just s-risk-specific work. And I think several of the arguments for humans doing some alignment work anyway apply to s-risk-specific work:
I would probably agree that alignment work is more likely to make a counterfactual difference to P(misalignment) than s-risk-targeted work is to make a counterfactual difference to P(s-risk), overall. But the gap seems to be overstated (and other prioritization considerations can outweigh this one, of course).
That post focuses on technical interventions, but a non-technical intervention that seems pretty hard to delegate to AIs is to reduce race dynamics between AI labs, which lead to an uncooperative multipolar takeoff.
I.e., the hardest part is ensuring the alignment of AIs on tasks that humans can't evaluate, where the ELK problem arises.
Claims about counterfactual value of interventions given AI assistance should be consistent
A common claim I hear about research on s-risks is that it’s much less counterfactual than alignment research, because if alignment goes well we can just delegate it to aligned AIs (and if it doesn’t, there’s little hope of shaping the future anyway).
I think there are several flaws with this argument that require more object-level context (see this post).[1] But at a high level, this consideration—that research/engineering can be delegated to AIs that pose little-to-no risk of takeover—should also make us discount the counterfactual value of alignment research/engineering. The main plan of OpenAI’s alignment team, and part of Anthropic’s plan and those of several thought leaders in alignment, is to delegate alignment work (arguably the hardest parts thereof)[2] to AIs.
I do in fact discount the counterfactual value of alignment for exactly this reason, BTW.
I would probably agree that alignment work is more likely to make a counterfactual difference to P(misalignment) than s-risk-targeted work is to make a counterfactual difference to P(s-risk), overall. But the gap seems to be overstated (and other prioritization considerations can outweigh this one, of course).
Agree with this point in particular.
Why I’m unconvinced by Tegmark’s argument for the mathematical universe hypothesis
The basic argument seems to be:
I don’t see why we should buy (2).
As far as I know, there are two arguments for (2). The first: “Every time our theories of physics have improved, we’ve dissolved what we thought was ‘stuff’ into abstract relations between more fundamental stuff. That is, we’ve dissolved intrinsic properties into relational properties.”
But our evidence is “we keep dissolving concepts of non-fundamental intrinsic properties into relations”, not “we keep dissolving all the intrinsic properties that those concepts cover”. E.g. when we learned that the concept of an “atom” is nothing over and above a relation between subatomic particles, we weren’t getting rid of any of the fundamental stuff that composes an atom. The claim that there’s no fundamental stuff to be found is metaphysical, not something we have inductive evidence for.
Which brings us to the second argument: “We never directly observe this intrinsic fundamental stuff. Physics only needs relational properties to explain our observations. By Occam’s razor, we should drop the stuff.”
But I don’t know what it even means to have relations without stuff (e.g. parts of spacetime) to be related — to have structure without substance. So our observations do seem to entail intrinsic properties. I’m open to the correct ontology being very counterintuitive. But compared to the cost of dropping an ontological view as basic as “relations need stuff for the relations to apply to”, the simplicity gains from doing so seem pretty weak.
You may be running two arguments together , there. Freedom from human independent concepts is supposed to support the idea of a mathematical universe, not a relational one. Then there is the claim that the idea that some , but only some, maths exists materially, is unnecessary baggage. Neither argument is totally convincing. It isn't obvious that maths isn't a human invention. It isn't obvious that an external world has to be independent of human concepts, as well as human imagination. It isn't obvious that maths can exist immaterially
Freedom from human independent concepts is supposed to support the idea of a mathematical universe, not a relational one
Tegmark says (p. 10 here) "the only intrinsic properties of a mathematical structure are its relations". So I think I am representing his actual argument for MUH in the OP. I think "the claim that the idea that some , but only some, maths exists materially, is unnecessary baggage" is meant to derive the Level IV multiverse from MUH.
It isn't obvious that maths isn't a human invention. It isn't obvious that an external world has to be independent of human concepts, as well as human imagination. It isn't obvious that maths can exist immaterially
FWIW I think I agree with all the non-obvious propositions here. I don't think they're the load-bearing premises of Tegmark's argument.
Linkpost: Why Evidential Cooperation in Large Worlds might not be action-guiding
A while back I wrote up why I was skeptical of ECL. I think this basically holds up, with the disclaimers at the top of the post. But I don't consider it that important compared to other things relevant to LW that people could be thinking about, so I decided to put it on my blog instead.
Is God's coin toss with equal numbers a counterexample to mrcSSA?
I feel confused as to whether minimal-reference-class SSA (mrcSSA) actually fails God's coin toss with equal numbers (where "failing" by my lights means "not updating from 50/50"):
In order words: It seems that the controversial setup in anthropics is in answering P(I [blah] | world), i.e., what we do when we introduce the indexical information about "I." But once we've picked out a particular "I," the different views should agree.
(I still feel suspicious of mrcSSA's metaphysics for independent reasons, but am considerably less confident in that than my verdict on God's coin toss with equal numbers.)
It seems that what I was missing here was: mrcSSA disputes my premise that the evidence in fact is "*I* am in a white room, [created by God in the manner described in the problem setup], and have a red jacket"!
Rather, mrcSSA takes the evidence to be: "Someone is in a white room, [created by God in the manner described in the problem setup], and has a red jacket." Which is of course certain to be the case given either heads or tails.
(h/t Jesse Clifton for helping me see this)
"Messy" tasks vs. hard-to-verify tasks
(Followup to here.)
I've found LLMs pretty useful for giving feedback on writing, including writing a fairly complex philosophical piece. Recently I wondered, "Hm, is this evidence that LLMs' capabilities can generalize well to hard-to-verify tasks? That would be an update for me (toward super-short timelines, for one)."
I haven't thought deeply about this yet, but I think the answer is: no. We should disentangle the intuitive “messiness” of the task of giving writing advice, from how difficult success is to verify:
Maybe this is obvious? My sense, though, is that these two things could easily get conflated.
“Reaction” is meant to capture all the stuff that makes someone consider some feedback “useful” immediately when (or not too long after) they read it. I’m not saying LLMs are trying to make me feel good about my writing in this context, though that’s probably true to some extent.