https://www.lesswrong.com/posts/DMxe4XKXnjyMEAAGw/the-geometric-expectation

2. Conversely, there is some (potentially high) threshold of societal epistemics + coordination + institutional steering beyond which we can largely eliminate anthropogenic x-risk, potentially in perpetuity

Note that this is not a logical converse of your first statement. I realize that the word "conversely" can be used non-strictly and might in fact be used this way by you here, but I'm stating this just in case.

My guess is that "there is some (potentially high) threshold of societal epistemics + coordination + institutional steering beyond which we can largely eliminate anthropogenic x-risk in perpetuity" is false — my guess is that improving [societal epistemics + coordination + institutional steering] is an infinite endeavor; I discuss this a bit here. That said, I think it is plausible that there is a possible position from which we could reasonably be fairly confident that things will be going pretty well for a really long time — I just think that this would involve one continuing to develop one's methods of [societal epistemics, coordination, institutional steering, etc.] as one proceeds.

Basically nobody actually wants the world to end, so if we do that to ourselves, it will be because somewhere along the way we weren’t good enough at navigating collective action problems, institutional steering, and general epistemics

... or because we didn't understand important stuff well enough in time (for example: if it is the case that by default, the first AI that could prove $P\ne NP$ would eat the Sun, we would want to firmly understand this ahead of time), or because we weren't good enough at thinking (for example, people could just be lacking in iq, or have never developed an adequate sense of what it is even like to understand something, or be intellectually careless), or because we weren't fast enough at disseminating or [listening to] the best individual understanding in critical cases, or because we didn't value the right kinds of philosophical and scientific work enough, or because we largely-ethically-confusedly thought some action would not end the world despite grasping some key factual broad strokes of what would happen after, or because we didn't realize we should be more careful, or maybe because generally understanding what will happen when you set some process in motion is just extremely cursed.^[1] I guess one could consider each of these to be under failures in general epistemics... but I feel like just saying "general epistemics" is not giving understanding its proper due here.

the long run equilibrium of the earth-originating civilization

(this isn’t centrally engaging with your shortform but:) it could be interesting to think about whether there will be some sort of equilibrium or development will meaningfully continue (until the heat death of the universe or until whatever other bound of that kind holds up or maybe just forever)^[1]

Summarizing documents, and exploring topics I'm no expert in: Super good

I think you probably did this, but I figured it's worth checking: did you check this on documents you understand well (such as your own writing) and topics you are an expert on?

I think this approach doesn't make sense. Issues, briefly:

• if you want to be squaring $D$, you need it to be square — you should append another row of $0$s
• this matrix $D$ does not have a logarithm, because it isn't invertible ( https://en.wikipedia.org/wiki/Logarithm_of_a_matrix#Existence )^[1]
• there in fact isn't any matrix $X$ that could reasonably be considered a $D^{3/2}$, because such an $X$ should satisfy $X^2=D^3$, but the matrix $D^3$ does not have a square root (see e.g. https://math.stackexchange.com/a/66156/540174 for how to think about this)

I'd rather your "that is" were a "for example". This is because:

• It's also possible for the process of updates to not be getting arbitrarily close to any endpoint (with a notion of closeness that is imo appropriate for this context), without there being any sentence on which one keeps changing one's mind. If we're thinking of one's "ethical state of mind" as being given by the probabilities one assigns to some given countable collection of sentences, then here I'm saying that it can be reasonable to use a notion of convergence which is stronger than pointwise convergence. For math, if one just runs a naive proof search and assigns truth value 1 to proven sentences and 0 to disproven sentences, one could try to say this sequence of truth value assignments is converging to the assignment that gives 1 to all provable sentences and 0 to all disprovable sentences (and whatever the initialization assigns to all independent sentences, let's say), but I think that in our context of imagining some long reflection getting close to something in finite time, it's more reasonable to say that one isn't converging to anything in this example — it seems pretty intuitive to say that after any finite number of steps, one hasn't really made much progress toward this kinda-endpoint (after all, one will have proved only finitely many things, and one still has infinitely many more things left to prove). Bringing this a tad closer to ethical reality: we could perhaps imagine someone repeatedly realizing that projects they hadn't really considered before are worth working on, infinitely many times, with what they are up to thus changing [by a lot] [infinitely many times]. To spell out the connection of this to the math example a bit more: the common point is that novelty can appear in the sentences/things considered, so one can have novelty even if novelty doesn't keep showing up in how one relates to any given sentence/thing. I say more about these themes here.

I feel like items on your current list have $<10\%$ of the responsibility for what I'd consider software updates in humans, and that it sorta fails to address almost all the ordinary stuff that goes on when individual humans are learning stuff (from others or independently) or when "humanity is improving its thinking". But that makes me think that maybe I'm missing what you're going for with this list?^[1] Continuing with the (possibly different) question I have in mind anyway, here's a list that imo points toward a decent chunk of what is missing from your list, with a focus on the case of independent and somewhat thoughtful learning/[thinking-improving] (as opposed to the case of copying from others, and as opposed to the case of fairly non-thoughtful thinking-improving)^[2]:

• a mathematician coming up with a good mathematical concept and developing a sense of how to use it (and ditto for a mathematical system/formalism)^[3]
• seeing a need to talk about something and coining a word for it
• a philosopher trying to clarify/re-engineer a concept, eg by seeing which more precise definition could accord with the concept having some desired "inferential role"
• noticing and resolving tensions in one’s views
• discovering/inventing/developing the scientific method; inventing/developing p-values; improving peer review
• discussing what kinds of evidence could help with some particular scientific question
• inventing writing; inventing textbooks
• the varied thought that is upstream of a professional poker player thinking the way they do when playing poker
• asking oneself "was that a reasonable inference?", “what auxiliary construction would help with this mathematical problem?”, "which techniques could work here?", "what is the main idea of this proof?", "is this a good way to model the situation?", "can I explain that clearly?", "what caused me to be confused about that?", "why did I spend so long pursuing this bad idea?", "how could I have figured that out faster?", “which question are we asking, more precisely?”, "why are we interested in this question?", “what is this analogous to?”, "what should I read to understand this better?", "who would have good thoughts on this?"

I agree it’s a pretty unfortunate/silly question. Searle’s analysis of it in Chapter 1 of Seeing Things as They Are is imo not too dissimilar to your analysis of it here, except he wouldn’t think that one can reasonably say “the world we see around us is an internal perceptual copy” (and I myself have trouble compiling this into anything true also), though he’d surely agree that various internal things are involved in seeing the world. I think a significant fraction of what’s going on with this “disagreement” is a bunch of “technical wordcels” being annoyed at what they consider to be careless speaking that they take to be somewhat associated with careless thinking.

see e.g. Chapter 1 of Searle's Seeing Things as They Are for an exposition of the view usually called direct realism (i'm pretty sure you guys (including the op) have something pretty different in mind than that view and i think it's plausible you all would actually just agree with that view)