I won't address why [AIs that humans create] might^[1] have their own alien values (so I won't address the "turning against us" part of your comment), but on these AIs outcompeting humans^[2]:

• There is immense demand for creating systems which do $\approx$anything better than humans, because there is demand for all the economically useful things humans do — if someone were to create such a thing and be able to control it, they'd become obscenely rich (and probably come to control the world^[3]).
• Also, it's possible to create systems that do $\approx$anything better than humans. In fact, it's probably not that hard — it'll probably happen at some point in this century by default (absent an AGI ban).

While I'm probably much more of a lib than you guys (at least in ordinary human contexts), I also think that people in AI alignment circles mostly have really silly conceptions of human valuing and the historical development of values.^[1] I touch on this a bit here. Also, if you haven't encountered it already, you might be interested in Hegel's work on this stuff — in particular, The Phenomenology of Spirit.

It's how science works: You focus on simple hypotheses and discard/reweight them according to Bayesian reasoning.

There are some ways in which solomonoff induction and science are analogous^[1], but there are also many important ways in which they are disanalogous. Here are some ways in which they are disanalogous:

• A scientific theory is much less like a program that prints (or predicts) an observation sequence than it is like a theory in the sense used in logic. Like, a scientific theory provides a system of talking which involves some sorts of things (eg massive objects) about which some questions can be asked (eg each object has a position and a mass, and between any pair of objects there is a gravitational force) with some relations between the answers to these questions (eg we have an axiom specifying how the gravitational force depends on the positions and masses, and an axiom specifying how the second derivative of the position relates to the force).^[2]
• Science is less in the business of predicting arbitrary observation sequences, and much more in the business of letting one [figure out]/understand/exploit very particular things — like, the physics someone knows is going to be of limited help when they try to predict the time sequence of intensities of pixel $(x,y)$ on their laptop screen, but it is going to help them a lot when solving the kinds of problems that would show up in a physics textbook.
• Even for solving problems that a theory is supposed to help one solve (and for the predictions it is supposed to help one make), a scientific theory is highly incomplete — in addition to the letter of the theory, a human solving the problems in a classical mechanics textbook will be majorly relying on tacit understanding gained from learning classical mechanics and their common-sense understanding.
• Making scientific progress looks less like picking out a correct hypothesis from some set of pre-well-specified hypotheses by updating on data, and much more like coming up with a decent way to think about something where there previously wasn't one. E.g. it could look like Faraday staring at metallic filings near a magnet and starting to talk about the lines he was seeing, or Lorentz, Poincaré, and Einstein making sense of the result of the Michelson-Morley experiment. Imo the bayesian conception basically completely fails to model gaining scientific understanding.
• Scientific theories are often created to do something — I mean: to do something other than predicting some existing data — e.g., to make something; e.g., see https://en.wikipedia.org/wiki/History_of_thermodynamics.
• Scientific progress also importantly involves inventing new things/phenomena to study. E.g., it would have been difficult to find things that Kirchhoff's laws could help us with before we invented electric circuits; ditto for lens optics and lenses).
• Idk, there is just very much to be said about the structure of science and scientific progress that doesn't show up in the solomonoff picture (or maaaybe at best in some cases shows up inexplicitly inside the inductor). I'll mention a few more things off the top of my head:
  • having multiple ways to think about something
  • creating new experimental devices/setups
  • methodological progress (e.g. inventing instrumental variable methods in econometrics)
  • mathematical progress (e.g. coming up with the notion of a derivative)
  • having a sense of which things are useful/interesting to understand
  • generally, a human scientific community doing science has a bunch of interesting structure; in particular, the human minds participating in it have a bunch of interesting structure; one in fact needs a bunch of interesting structure to do science well; in fact, more structure of various kinds is gained when making scientific progress; basically none of this is anywhere to be seen in solomonoff induction

However, the reference class that includes the theory of computation is one possible reference class that might include the theory of agents.[1] But for all (I think) we know, the reference class we are in might also be (or look more like) complex systems studies, where you can prove a bunch of neat things, but there's also a lot of behavior that is not computationally reducible and instead you need to observe, simulate, crunch the numbers. Moreover, noticing surprising real-world phenomena can serve as a guide to your attempts to explain the observed phenomena in ~mathematical terms (e.g., how West et al. explained (or re-derived) Kleiber's law from the properties of intra-organismal resource supply networks[2]). I don't know what the theory will look like; to me, its shape remains an open a posteriori question.

along an axis somewhat different than the main focus here, i think the right picture is: there is a rich field of thinking-studies. it’s like philosophy, math, or engineering. it includes eg Chomsky's work on syntax, Turing’s work on computation, Gödel’s work on logic, Wittgenstein’s work on language, Darwin's work on evolution, Hegel’s work on development, Pascal’s work on probability, and very many more past things and very many more still mostly hard-to-imagine future things. given this, i think asking about the character of a “theory of agents” would already soft-assume a wrong answer. i discuss this here

i guess a vibe i'm trying to communicate is: we already have thinking-studies in front of us, and so we can look at it and get a sense of what it's like. of course, thinking-studies will develop in the future, but its development isn't going to look like some sort of mysterious new final theory/science being created (though there will be methodological development (like for example the development of set-theoretic foundations in mathematics, or like the adoption of statistics in medical science), and many new crazy branches will be developed (of various characters), and we will surely $\approx$resolve various particular questions in various ways (though various other questions call for infinite investigations))

Hmm, thanks for telling me, I hadn't considered that. I think I didn't notice this in part because I've been thinking of the red-black circle as being "canceled out"/"negated" on the flag, as opposed to being "asserted". But this certainly wouldn't be obvious to someone just seeing the flag.

I designed a pro-human(ity)/anti-(non-human-)AI flag:

• The red-black circle is HAL's eye; it represents the non-human in-all-ways-super-human AI(s) that the world's various AI capability developers are trying to create, that will imo by default render all remotely human beings completely insignificant and cause humanity to completely lose control over what happens :(.
• The white star covering HAL's eye has rays at the angles of the limbs of Leonardo's Vitruvian Man; it represents humans/humanity remaining more capable than non-human AI (by banning AGI development and by carefully self-improving).
• The blue background represents our potential self-made ever-better future, involving global governance/cooperation/unity in the face of AI.

Feel free to suggest improvements to the flag. Here's latex to generate it:

% written mostly by o3 and o4-mini-high, given k's prompting
% an anti-AI flag. a HAL "eye" (?) is covered by a vitruvian man star
\documentclass[tikz]{standalone}
\usetikzlibrary{calc}
\usepackage{xcolor}                 % for \definecolor
\definecolor{UNBlue}{HTML}{5B92E5}

\begin{document}
\begin{tikzpicture}
%--------------------------------------------------------
% flag geometry
%--------------------------------------------------------
\def\flagW{6cm}     % width  -> 2 : 3 aspect
\def\flagH{4cm}     % height
\def\eyeR {1.3cm}     % HAL-eye radius

% light-blue background
\fill[UNBlue] (0,0) rectangle (\flagW,\flagH);

%--------------------------------------------------------
% concentric “HAL eye” (outer-most ring first)
%--------------------------------------------------------
\begin{scope}[shift={(\flagW/2,\flagH/2)}] % centre of the flag
 \foreach \f/\c in {%
     1.00/black,
     .68/{red!50!black},
     .43/{red!80!orange},
     .1/orange,
     .05/yellow}%
 {%
   \fill[fill=\c,draw=none] circle ({\f*\eyeR});
 }

%── parameters ───────────────────────────────────────
\def\R{\eyeR}        % distance from centre to triangle’s tip
\def\Alpha{10}       % full apex angle (°)
%── compute half-angle & half-base once ─────────────
\pgfmathsetmacro\halfA{\Alpha/2}               
\pgfmathsetlengthmacro\halfside{\R*tan(\halfA)}

%── loop over Vitruvian‐man angles ───────────────────
\foreach \Beta in {0,30,90,150,180,240,265,275,300} {%
 % apex on the eye‐rim
 \coordinate (A) at (\Beta:\R);
 % base corners offset ±90°
 \coordinate (B) at (\Beta+90:\halfside);
 \coordinate (C) at (\Beta-90:\halfside);
 % fill the spike
 \path[fill=white,draw=none] (A) -- (B) -- (C) -- cycle;
}

\end{scope}
\end{tikzpicture}
\end{document}
 

https://www.lesswrong.com/posts/DMxe4XKXnjyMEAAGw/the-geometric-expectation

2. Conversely, there is some (potentially high) threshold of societal epistemics + coordination + institutional steering beyond which we can largely eliminate anthropogenic x-risk, potentially in perpetuity

Note that this is not a logical converse of your first statement. I realize that the word "conversely" can be used non-strictly and might in fact be used this way by you here, but I'm stating this just in case.

My guess is that "there is some (potentially high) threshold of societal epistemics + coordination + institutional steering beyond which we can largely eliminate anthropogenic x-risk in perpetuity" is false — my guess is that improving [societal epistemics + coordination + institutional steering] is an infinite endeavor; I discuss this a bit here. That said, I think it is plausible that there is a possible position from which we could reasonably be fairly confident that things will be going pretty well for a really long time — I just think that this would involve one continuing to develop one's methods of [societal epistemics, coordination, institutional steering, etc.] as one proceeds.

Basically nobody actually wants the world to end, so if we do that to ourselves, it will be because somewhere along the way we weren’t good enough at navigating collective action problems, institutional steering, and general epistemics

... or because we didn't understand important stuff well enough in time (for example: if it is the case that by default, the first AI that could prove $P\ne NP$ would eat the Sun, we would want to firmly understand this ahead of time), or because we weren't good enough at thinking (for example, people could just be lacking in iq, or have never developed an adequate sense of what it is even like to understand something, or be intellectually careless), or because we weren't fast enough at disseminating or [listening to] the best individual understanding in critical cases, or because we didn't value the right kinds of philosophical and scientific work enough, or because we largely-ethically-confusedly thought some action would not end the world despite grasping some key factual broad strokes of what would happen after, or because we didn't realize we should be more careful, or maybe because generally understanding what will happen when you set some process in motion is just extremely cursed.^[1] I guess one could consider each of these to be under failures in general epistemics... but I feel like just saying "general epistemics" is not giving understanding its proper due here.

the long run equilibrium of the earth-originating civilization

(this isn’t centrally engaging with your shortform but:) it could be interesting to think about whether there will be some sort of equilibrium or development will meaningfully continue (until the heat death of the universe or until whatever other bound of that kind holds up or maybe just forever)^[1]