I found this post valuable for stating an (apparently popular on LW) coherentist view pretty succinctly. But its dismissal of foundationalism seems way too quick:

• It's not at all "pretty clear" to me that "any foundations are also subject to justificatory work". Foundationalist philosophers don’t pick arbitrary beliefs (or seemings/appearances/etc.) to call foundational. They try to identify foundations that seem intuitively to (1) not be the sort of thing that requires justification, and (2) provide justification for non-foundational beliefs.
  • Fair enough if one doesn't find the arguments for such foundations persuasive. But I would think one should engage at least a bit with those arguments in a post like this.
• There seems to be a misunderstanding of what foundationalism entails, in this quote: "We're not going to find a set of axioms which just seem obvious to all humans once articulated. Rather, there's some work to be done to make them seem obvious." Foundationalists don't deny this — something can be foundational in the sense of requiring no further justification when you fully understand it, without being obvious before you fully understand it.^[1]

1. ^{^}

   Relevant quote from Nye (2015): “The methodological approach I defend maintains that the direct plausibility or implausibility of principles about the ethical relevance of various factors is foundational in normative and practical ethics. This does not mean that appearances of direct plausibility are infallible. Principles often seem plausible only because we are making confusions and do not fully appreciate what they are really saying. On the approach I defend, much of the business of ethical reasoning consists in correcting erroneous appearances of plausibility by clarifying the content of principles, making crucial distinctions, and discovering alternatives with greater direct plausibility. Nor does the claim that the direct plausibility of principles is foundational mean that we should begin our ethical reasoning by considering only which principles seem plausible. The principles that turn out to be most plausible on reflection might be suggested to us only by first considering our intuitions about a variety of cases and then seeing which of them can be justified by principles that, once formulated and clarified, are directly plausible.”

Your example would be pretty telling and damning if we assume that you're correct, but my guess is that most readers here will assume you're wrong about it. Someone in your position could still be right, of course; I'm just saying that this wouldn't yet be apparent to readers.

Fair enough! :) The parallel I had in mind was "[almost] no object level pushback", or at least almost no object level pushback that I can tell is based on an accurate understanding of my arguments.

So far, I genuinely have not gotten much object-level pushback on the most load-bearing points of my sequence, so, I'm not that worried.

I think this probably underestimates the severity of founder effects in this community. A salient example to me is precise Bayesianism (and the cluster of epistemologies that try to "approximate" Bayesianism): As far as I can tell, the rationalist and EA communities went over a decade without anyone within these communities pushing back on the true weak points of this epistemology, which is extremely load-bearing for cause prioritization. I think in hindsight we should have been worried about missing this sort of thing.

Examples of awareness growth vs. logical updates

(Thanks to Lukas Finnveden for discussion that prompted these examples, and for authoring examples #3-#6 verbatim.)

A key concept in the theory of open-minded updatelessness (OMU) is "awareness growth", i.e., conceiving of hypotheses you hadn't considered before. It's helpful to gesture at "discovering crucial considerations" as examples of awareness growth. But not all CC discoveries are awareness growth. And we might think we don't need this OMU idea if awareness growth is just logical updating, i.e. you already had nonzero credence in some hypothesis, but you changed this credence purely by thinking more. What's the difference? Here are some examples.

1. I realize it's possible that I’m in a simulation.
   • Awareness growth. Because before realizing that, I simply hadn’t conceived of “I’m in a sim” as a way the world could be. (I don’t know how/why I could/should model myself-before-I-read-Bostrom as having had nonzero credence in “I’m in a sim”.)
2. I realize that the simulation argument implies (under such and such assumptions) I should have high credence in "I’m in a simulation".
   • Logical update. Because I’m not discovering a new way the world could be, I’m just learning of a logical implication governing my credences over ways the world could be.
3. Someone has conceived of the idea that they might be in a simulation ("like a video game!") but hasn't considered that simulations might be done for scientific investigations of the past. This updates their view on how likely it is that they're in a simulation, and also makes them think that more of the probability mass goes to the specific type "investigation of the past"-sim.
   • The first sentence is awareness growth "by refinement" (Steele and Stefansson (2021, Sec 3.3.2)). The more specific hypotheses you’re now aware of were covered by a more coarse hypothesis you were previously aware of. And the second sentence is a logical update.
4. I see a rainbow colored car. I had never previously explicitly pictured a rainbow colored car, or thought about those words.
   • Pretty straightforward awareness growth.
5. I'm forecasting the likelihood of regime change in a country. I look up the base rates. I forecast based on them and some inside-view adjustments. Later on, the news come in that there was a regime change due to chaos from a natural disaster. I hadn't conceptualized that possible contribution to regime change! But fortunately I had implicitly accounted for it via base rates that (it turns out) included some examples of natural disasters.
   • Also pretty straightforward awareness growth. It's just not the decision-relevant kind, to the extent that you already implicitly priced in “natural disasters lead to regime change” via the base rate. (I think in practice it will very often be ambiguous how precisely we’re pricing in hypotheses we’re unaware of, even if one might argue we’re always kinda sorta pricing them in if you squint.)
6. Someone learns Newtonian physics.
   • A mix of awareness growth and a logical update. E.g., the exact law “F = ma” wouldn’t have occurred to most people before learning any Newtonian physics — that's awareness growth. But some people might've both (a) conceived of an object getting pushed in a vacuum never slowing down, yet (b) not assigned very high P(object never slows down | object is pushed in a vacuum) before learning this was a law — that's a logical update.

tendency to "bite bullets" or accepting implications that are highly counterintuitive to others or even to themselves, instead of adopting more uncertainty

I find this contrast between "biting bullets" and "adopting more uncertainty" strange. The two seem orthogonal to me, as in, I've ~just as frequently (if not more often) observed people overconfidently endorse their pretheoretic philosophical intuitions, in opposition to bullet-biting.

What other, perhaps slightly more complex or less obvious, crucial considerations are we still missing?

I agree this is very important. I've argued that if we appropriately price in missing crucial considerations,^[1] we should consider ourselves clueless about AI risk interventions (here and here).

1. ^{^}

   Also relatively prosaic causal pathways we haven't thought of in detail, not just high-level "considerations" per se.

A salient example to me: This post essentially consists of Paul briefly remarking on some mildly interesting distinctions about different kinds of x-risks, and listing his precise credences without any justification for them. It's well-written for what it aims to be (a quick take on personal views), but I don't understand why this post was so strongly celebrated.

I'm curious if you think you could have basically written this exact post a year ago. Or if not, what's the relevant difference? (I admit this is partly a rhetorical question, but it's mostly not.)

Oops, right. I think what's going on is:

• "It's only permissible to bet at odds that are inside your representor" is only true if the representor is convex. If my credence in some proposition X is, say, P(X) = (0.2, 0.49) U (0.51, 0.7), IIUC it's permissible to bet at 0.5. I guess the claim that's true is "It's only permissible to bet at odds in the convex hull of your representor".
• But I'm not aware of an argument that representors should be convex in general.
  • If there is such an argument, my guess is that the way things would work is: We start with the non-convex set of distributions that seem no less reasonable than each other, and then add in whichever other distributions are needed to make it convex. But there would be no particular reason we'd need to interpret these other distributions as "reasonable" precise beliefs, relative to the distributions in the non-convex set we started with.
• And, the kind of precise distribution P that would rationalize e.g. working on shrimp welfare seems to be the analogue of "betting at 0.5" in my example above. That is:
  • Our actual "set of distributions that seem no less reasonable than each other" would include some distributions that imply large positive long-term EV from working on shrimp welfare, and some that imply large negative long-term EV.
  • Whereas the distributions like P that imply vanishingly small long-term EV — given any evidence too weak to resolve our cluelessness w.r.t. long-term welfare — would lie in the convex hull. So betting at odds P would be permissible, and yet this wouldn't imply that P is "reasonable" as precise beliefs.

Sorry, I don't understand the argument yet. Why is it clear that I should bet on odds P, e.g., if P is the distribution that the CCT says I should be represented by?