This is a special post for quick takes by Anthony DiGiovanni. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I continue to be puzzled as to why many people on LW are very confident in the "algorithmic ontology" about decision theory:
So I see all axes except the "algorithm" axis as "live debates" -- basically anyone who has thought about it very much seems to agree that you control "the policy of agents who sufficiently resemble you" (rather than something more myopic like "your individual action")
Can someone point to resources that clearly argue for this position? (I don't think that, e.g., the intuition that you ought to cooperate with your exact copy in a Prisoner's Dilemma — much as I share it — is an argument for this ontology. You could endorse the physicalist ontology + EDT, for example.)
I can't point you to existing resources, but from my perspective, I assumed an algorithmic ontology because it seemed like the only way to make decision theory well defined (at least potentially, after solving various open problems). That is, for an AI that knows its own source code S, you could potentially define the "consequences of me doing X" as the logical consequences of the logical statement "S outputs X". Whereas I'm not sure how this could even potentially be defined under a physicalist ontology, since it seems impossible for even an ASI to know the exact details of itself as a physical system.
This does lead to the problem that I don't know how to apply LDT to humans (who do not know their own source code), which does make me somewhat suspicious that the algorithmic ontology might be a wrong approach (although physicalist ontology doesn't seem to help). I mentioned this as problem #6 in UDT shows that decision theory is more puzzling than ever.
ETA: I was (and still am) also under strong influence of Tegmark's Mathematical universe hypothesis. What's your view on it?
Thanks, that's helpful!
I am indeed interested in decision theory that applies to agents other than AIs that know their own source code. Though I'm not sure why it's a problem for the physicalist ontology that the agent doesn't know the exact details of itself — seems plausible to me that "decisions" might just be a vague concept, which we still want to be able to reason about under bounded rationality. E.g. under physicalist EDT, what I ask myself when I consider a decision to do X is, "What consequences do I expect conditional on my brain-state going through the process that I call 'deciding to do X' [and conditional on all the other relevant info I know including my own reasoning about this decision, per the Tickle Defense]?" But I might miss your point.
Re: mathematical universe hypothesis: I'm pretty unconvinced, though I at least see the prima facie motivation (IIUC: we want an explanation for why the universe we find ourselves in has the dynamical laws and initial conditions it does, rather than some others). Not an expert here, this is just based on some limited exploration of the topic. My main objections:
Not sure what you mean by "the math" exactly. I've heard people cite the algorithmic ontology as a motivation for, e.g., logical updatelessness, or for updateless decision theory generally. In the case of logical updatelessness, I think (low confidence!) the idea is that if you don't see yourself as this physical object that exists in "the real world," but rather see yourself as an algorithm instantiated in a bunch of possible worlds, then it might be sensible to follow a policy that doesn't update on e.g. the first digit of pi being odd.
query rephrase: taboo both "algorithmic ontology" and "physicalist ontology". describe how each of them constructs math to describe things in the world, and how that math differs. That is, if you're saying you have an ontology, presumably this means you have some math and some words describing how the math relates to reality. I'm interested in a comparison of that math and those words; so far you're saying things about a thing I don't really understand as being separate from physicalism. Why can't you just see yourself as multiple physical objects and still have a physicalist ontology? what makes these things different in some, any, math, as opposed to only being a difference in how the math connects to reality?
I think I just don't understand / probably disagree with the premise of your question, sorry. I'm taking as given whatever distinction between these two ontologies is noted in the post I linked. These don't need to be mathematically precise in order to be useful concepts.
ah my bad, my attention missed the link! that does in fact answer my whole question, and if I hadn't missed it I'd have had nothing to ask :)
There isn't a difference, though it's only because algorithms can simulate all of the physical stuff that's in the physical ontology, and thus the physical ontology is a special case of the algorithmic ontology.
Helpful link here to build intuition:
http://www.amirrorclear.net/academic/ideas/simulation/index.html
That's what I already believed, but OP seems to disagree, so I'm trying to understand what they mean
The argument for this is spelled out in Eliezer and Nate's Functional Decision Theory: A New Theory of Instrumental Rationality. See also the LessWrong wiki tag page
Thanks — do you have a specific section of the paper in mind? Is the idea that this ontology is motivated by "finding a decision theory that recommends verdicts in such and such decision problems that we find pre-theoretically intuitive"?
That sounds like a good description of my understanding, but I'd also say the pre-theoretic intuitions are real damn convincing!
There's a table of contents which you can use to read relevant sections of the paper. You know your cruxes better than I do.
shrug — I guess it's not worth rehashing pretty old-on-LW decision theory disagreements, but: (1) I just don't find the pre-theoretic verdicts in that paper nearly as obvious as the authors do, since these problems are so out-of-distribution. Decision theory is hard. Also, some interpretations of logical decision theories give the pre-theoretically "wrong" verdict on "betting on the past." (2) I pre-theoretically find the kind of logical updatelessness that some folks claim follows from the algorithmic ontology pretty bizarre. (3) On its face it seems more plausible to me that algorithms just aren’t ontologically basic, they’re abstractions we use to represent (physical) input-output processes.
From my perspective, I tend to think that the algorithmic ontology is better for decision theory in general because it's a strictly more general ontology than physicalism, with the physicalist ontology, including all of it's objects and models existing in the algorithmic ontology as a special case of the more general algorithmic ontology, so from a general perspective, it's better to work in the ontology that works in more cases than a physicalist ontology, while remembering that the physicalist ontology can exist and be valid as a special case of the algorithmic ontology.
I don't understand. It seems that when people appeal to the algorithmic ontology to motivate interesting decision-theoretic claims — like, say, "you should choose to one-box in Transparent Newcomb" — they're not just taking a more general perspective. They're making a substantive claim that it's sensible to regard yourself as an algorithm, over and above your particular instantiation in concrete reality.
Another way to say it is that the substantive part of the claim around decision theory that favors the algorithmic ontology is that your identity is closer to an isomorphism class/equivalence class of programs than a concrete instantiation of a program, akin to generic programming:
https://en.wikipedia.org/wiki/Generic_programming
https://en.wikipedia.org/wiki/Parametric_polymorphism
An argument for that is in an infinite universe where our patch of the affectable universe isn't special at the large scale in how it clumps up atoms into bigger structures, which is very probable then if you had arbitrarily good faster-than-light travel, you'd eventually meet copies of yourself, since there's only a finite number of possibilities for your identity, due to finite memory (conditional on you existing), but while it's combinatorially large, it's not infinite, and thus some states have to be repeated in an infinite universe, so your identity isn't unique under the instance-focused worldview (because there are infinite instances, not 1 instance), but from the algorithmic perspective they are one you, and thus you can trade with yourself.
The more practical argument is that a lot of the algorithmic ontology stuff was focused on future people who are closer to AIs today (but better), in that they can merge/copy to the extent that they have compute, and importantly are way more parallel in the sense that they can meaningfully have multiple experiences like current AI probably does today, and thus a lot of the issue with translating it to the human case is that humans are way more serial than AI.
you'd eventually meet copies of yourself
But a copy of me =/= me. I don't see how you establish this equivalence without assuming the algorithmic ontology in the first place.
Okay, my point here was that the copies would not only have the same algorithm, but also the same physical structure arbitrarily finely, and I don't need to assume the algorithmic ontology, I only need to remember that there's only a finite amount of configurations of atoms that end up in human bodies, meaning that the number of distinct identities is upper bounded by a finite number. The search space is combinatorically large, but not infinitely large, which ensures that in an infinite universe, some states must be repeated exactly.
That's why you'd meet yourself, eventually, it's not because of the algorithmic ontology, but because there isn't an infinite number of possibilities for your identity.
the copies would not only have the same algorithm, but also the same physical structure arbitrarily finely
I understand, I'm just rejecting the premise that "same physical structure" implies identity to me. (Perhaps confusingly, despite the fact that I'm defending the "physicalist ontology" in the context of this thread (in contrast to algorithmic ontology), I reject physicalism in the metaphysics sense.)
This also seems tangential, though, because the substantive appeals to the algorithmic ontology that get made in the decision theory context aren't about physically instantiated copies. They're about non-physically-instantiated copies of your algorithm. I unfortunately don't know of a reference for this off the top of my head, but it has come up in some personal communications FWIW.
Examples of awareness growth vs. logical updates
(Thanks to Lukas Finnveden for discussion that prompted these examples, and for authoring examples #3-#6 verbatim.)
A key concept in the theory of open-minded updatelessness (OMU) is "awareness growth", i.e., conceiving of hypotheses you hadn't considered before. It's helpful to gesture at "discovering crucial considerations" as examples of awareness growth. But not all CC discoveries are awareness growth. And we might think we don't need this OMU idea if awareness growth is just logical updating, i.e. you already had nonzero credence in some hypothesis, but you changed this credence purely by thinking more. What's the difference? Here are some examples.
Isn't the "you get what you measure" problem a problem for capabilities progress too, not just alignment? I.e.: Some tasks are sufficiently complex (hence hard to evaluate) and lacking in unambiguous ground-truth feedback that, when you turn the ML crank on them, you're not necessarily going to select for actually doing the task well. You'll select for "appearing to do the task well," and it's open question how well this correlates with actually doing the task well. ("Doing the task" here can include something much higher-level, like "being 'generally intelligent'.")
Which isn't to say this problem wouldn't bite especially hard for alignment. Alignment seems harder to verify than lots of things. But this is one reason I'm not fully sold that once you get human-level AI, capabilities progress will get faster.
(I'm hardly an expert on this, so might well have missed existing discourse on & answers to this question.)
Linkpost: "Against dynamic consistency: Why not time-slice rationality?"
This got too long for a "quick take," but also isn't polished enough for a top-level post. So onto my blog it goes.
I’ve been skeptical for a while of updateless decision theory, diachronic Dutch books, and dynamic consistency as a rational requirement. I think Hedden's (2015) notion of time-slice rationality nicely grounds the cluster of intuitions behind this skepticism.
Endorsement on reflection is not straightforward, even states of knowledge or representations of values or ways of interpreting them can fail to be endorsed. It's not good from my perspective for someone else to lose themselves and start acting in my interests. But it is good for them to find themselves if they are confused about what they should endorse on reflection.
From 0-my perspective, it’s good for 1-me to believe updatelessness is rational, even if from 1-my perspective it isn’t.
Values can say things about how agents think, about the reasons behind outcomes, not just the outcomes themselves. An object level moral point that gestures at the issue is to say that it's not actually good when a person gets confused or manipulated and starts working towards an outcome that I prefer, that is I don't prefer an outcome when it's bundled with a world that produced it in this way, even if I would prefer the outcome when considered on its own. So I disagree with the claim that, assuming "from 1-my perspective it's not good to do X", then it's still "from 0-my perspective it's good for 1-me to believe that they should do X".
A metaethical point on interpretation of states of knowledge or values not being straightforward is about the nature of possible confusion about what an agent might value. There is a setting where decision theory is sorted out, and values are specified explicitly, so that the notion of them being confused is not under consideration. But if we do entertain the possibility of confusion, that the design isn't yet settled, or that there is no reflective stability, then the thing that's currently written down as "values" and determines immediate actions has little claim to be actual values.
Claims about counterfactual value of interventions given AI assistance should be consistent
A common claim I hear about research on s-risks is that it’s much less counterfactual than alignment research, because if alignment goes well we can just delegate it to aligned AIs (and if it doesn’t, there’s little hope of shaping the future anyway).
I think there are several flaws with this argument that require more object-level context (see this post).[1] But at a high level, this consideration—that research/engineering can be delegated to AIs that pose little-to-no risk of takeover—should also make us discount the counterfactual value of alignment research/engineering. The main plan of OpenAI’s alignment team, and part of Anthropic’s plan and those of several thought leaders in alignment, is to delegate alignment work (arguably the hardest parts thereof)[2] to AIs.
It’s plausible (and apparently a reasonably common view among alignment researchers) that:
It seems that if these claims hold, lots of alignment work would be made obsolete by AIs, not just s-risk-specific work. And I think several of the arguments for humans doing some alignment work anyway apply to s-risk-specific work:
I would probably agree that alignment work is more likely to make a counterfactual difference to P(misalignment) than s-risk-targeted work is to make a counterfactual difference to P(s-risk), overall. But the gap seems to be overstated (and other prioritization considerations can outweigh this one, of course).
That post focuses on technical interventions, but a non-technical intervention that seems pretty hard to delegate to AIs is to reduce race dynamics between AI labs, which lead to an uncooperative multipolar takeoff.
I.e., the hardest part is ensuring the alignment of AIs on tasks that humans can't evaluate, where the ELK problem arises.
Claims about counterfactual value of interventions given AI assistance should be consistent
A common claim I hear about research on s-risks is that it’s much less counterfactual than alignment research, because if alignment goes well we can just delegate it to aligned AIs (and if it doesn’t, there’s little hope of shaping the future anyway).
I think there are several flaws with this argument that require more object-level context (see this post).[1] But at a high level, this consideration—that research/engineering can be delegated to AIs that pose little-to-no risk of takeover—should also make us discount the counterfactual value of alignment research/engineering. The main plan of OpenAI’s alignment team, and part of Anthropic’s plan and those of several thought leaders in alignment, is to delegate alignment work (arguably the hardest parts thereof)[2] to AIs.
I do in fact discount the counterfactual value of alignment for exactly this reason, BTW.
I would probably agree that alignment work is more likely to make a counterfactual difference to P(misalignment) than s-risk-targeted work is to make a counterfactual difference to P(s-risk), overall. But the gap seems to be overstated (and other prioritization considerations can outweigh this one, of course).
Agree with this point in particular.
Linkpost: Why Evidential Cooperation in Large Worlds might not be action-guiding
A while back I wrote up why I was skeptical of ECL. I think this basically holds up, with the disclaimers at the top of the post. But I don't consider it that important compared to other things relevant to LW that people could be thinking about, so I decided to put it on my blog instead.
Is God's coin toss with equal numbers a counterexample to mrcSSA?
I feel confused as to whether minimal-reference-class SSA (mrcSSA) actually fails God's coin toss with equal numbers (where "failing" by my lights means "not updating from 50/50"):
In order words: It seems that the controversial setup in anthropics is in answering P(I [blah] | world), i.e., what we do when we introduce the indexical information about "I." But once we've picked out a particular "I," the different views should agree.
(I still feel suspicious of mrcSSA's metaphysics for independent reasons, but am considerably less confident in that than my verdict on God's coin toss with equal numbers.)
It seems that what I was missing here was: mrcSSA disputes my premise that the evidence in fact is "*I* am in a white room, [created by God in the manner described in the problem setup], and have a red jacket"!
Rather, mrcSSA takes the evidence to be: "Someone is in a white room, [created by God in the manner described in the problem setup], and has a red jacket." Which is of course certain to be the case given either heads or tails.
(h/t Jesse Clifton for helping me see this)
"Messy" tasks vs. hard-to-verify tasks
(Followup to here.)
I've found LLMs pretty useful for giving feedback on writing, including writing a fairly complex philosophical piece. Recently I wondered, "Hm, is this evidence that LLMs' capabilities can generalize well to hard-to-verify tasks? That would be an update for me (toward super-short timelines, for one)."
I haven't thought deeply about this yet, but I think the answer is: no. We should disentangle the intuitive “messiness” of the task of giving writing advice, from how difficult success is to verify:
Maybe this is obvious? My sense, though, is that these two things could easily get conflated.
“Reaction” is meant to capture all the stuff that makes someone consider some feedback “useful” immediately when (or not too long after) they read it. I’m not saying LLMs are trying to make me feel good about my writing in this context, though that’s probably true to some extent.