1. Some people believe that (a) controlled on-paradigm ASI is possible, but that (b) it would require spending some nontrivial amount of resources/time on alignment/control research^[1], and that (c) the US AGI labs are much more likely to do it than the Chinese ones. Therefore, the US winning is less likely to lead to omnicide.
   • I think it's not unreasonable to believe (c), so if you believe (a) and (b), as many people do, the conclusion checks out. I assign low (but nonzero) probability to (a), though.
2. Even if the Chinese labs can keep ASI aligned/under control, some people are scared of being enslaved by the CCP, and think that the USG becoming god is going to be better for them.^[2] This probably includes people who profess to only care about the nobody-should-build-it thing: they un/semi-consciously track the S-risk possibility, and it's awful-feeling enough to affect their thinking even if they assign it low probability.
   • I think that's a legitimate worry; S-risks are massively worse than X-risks. But I don't expect the USG's apotheosis to look pretty either, especially not under the current administration, and same for the apotheosis of most AGI labs, so the point is mostly moot.
   • I guess Anthropic or maybe DeepMind could choose non-awful results? So sure, if the current paradigm can lead to controlled ASI, and the USG stays asleep, and Anthropic/DM are the favorites to win, "make China lose" has some sense.
3. Variant on the above scenarios, but which does involve an international pause, with some coordination to only develop ASI once it can be kept under control. This doesn't necessarily guarantee that the ASI, once developed, will be eudaimonic, so "who gets to ASI first/has more say on ASI" may matter; GOTO (2).
4. The AI-Risk advocates may feel that they have more influence on the leadership of the US labs. For US-based advocates, this is almost certainly correct. If that leadership can be convinced to pause, this buys us as much time as it'd take for the runners-up to catch up. Thus, the further behind China is, the more time we can buy in this hypothetical.
   1. In addition, if China is way behind, it's more likely that the US AGI labs would agree to stop, since more time to work would increase the chances of success of [whatever we want the pause for, e. g. doing alignment research or trying to cause an international ban].
5. Same as (4), but for governments. Perhaps the USG is easier to influence into arguing for an international pause. If so, both (a) the USG is more likely to do this if it feels that it's comfortably ahead of China rather than nose-to-nose, (b) China is more likely to agree to an international ban if the USG is speaking from a position of power and is ahead on AI than if it's behind/nose-to-nose. (Both because the ban would be favorable to China geopolitically, and because the X-risk arguments would sound more convincing if they don't look like motivated reasoning/bullshit you're inventing to convince China to abandon a technology that gives it geopolitical lead on the US.)
6. Some less sensible/well-thought-out variants of the above, e. g.:
   • Having the illusion of having more control over the US labs/government.
   • Semi/un-consciously feeling that it'd be better if your nation ends the world than if the Chinese do it.
   • Semi/un-consciously feeling that it'd be better if your nation is more powerful/ahead of a foreign one, independent of any X-risk considerations.
7. Suppose you think the current paradigm doesn't scale to ASI, or that we'll succeed in internationally banning ASI research. The amount of compute at a nation's disposal is likely to still be increasingly important in the coming future (just because it'd allow to better harness the existing AI technology, for military and economic ends). Thus, constricting China's access is likely to be better for the US as well.
   • This has nothing to do with X-risks though, it's prosaic natsec stuff.

(The post describes a fallacy where you rule out a few specific members of a set using properties specific to those members, and proceed to conclude that you've ruled out that entire set, having failed to consider that it may have other members which don't share those properties. My comment takes specific examples of people falling into this fallacy that happened to be mentioned in the post, rules out that those specific examples apply to me, and proceeds to conclude that I'm invulnerable to this whole fallacy, thus committing this fallacy.

(Unless your comment was intended to communicate "I think your joke sucks", which, valid.))

We should, of course, assume that the algorithm computing our universe is simple to describe in terms of the "natural" reference language, due to the simplicity prior. I. e., it should have support for the basic functions our universe's physics computes. I think that's already equivalent to "the machine can run our physics without insane implementation size".

On the flip side, it's allowed to lack support for functions our universe can't cheaply compute. For example, it may not have primitive functions for solving NP-complete problems. (In theory, I think there was nothing stopping physics from having fundamental particles that absorb Traveling Salesman problems and near-instantly emit their solutions.)

Now suppose we also assume that our observations are sampled from the distribution over all observers in Tegmark 4. This means that when we're talking about the language/TM underlying it, we're talking about some "natural", "objective" reference language.

What can we infer about it?

First, as mentioned, we should assume the reference language is not a Turing tarpit. After all, if we allowed reality to "think" in terms of some arbitrarily convoluted Turing-tarpit language, we could arbitrarily skew the simplicity prior.

But what is a "Turing tarpit" in that "global"/"objective" sense, not defined relative to some applications/programs? Intuitively, it feels like "one of the normal, sane languages that could easily implement all the other sane languages" should be possible to somehow formalize...

Which is to say: when we're talking about the Kolmogorov complexity of some algorithm, in what language are we measuring it? Intuitively, we want to, in turn, pick one of the "simplest" languages to define.^[1] But what language do we pick for measuring this language's complexity? An infinite recursion follows.

Intuitively, there's perhaps some way to short-circuit that recursion. (Perhaps by somehow defining the complexity of a language by weighing its complexity across "all" languages while prioritizing the opinions of those languages which are themselves simple in terms of whatever complexity measure this expression defines? Or something along those lines, circular definitions not always a problem. (Though see an essay Tsvi linked to which breaks down why many of those definitions don't work.))

Regardless, if something like this is successful, we'll get a "global" definition of what counts as a simple/natural language. This would, in turn, allow us to estimate the "objective" complexity of various problems, by measuring the length of their solutions in terms of that natural language (i. e., the length of the execution trace of a computation solving the problem). This would perhaps show that some problems are "objectively" hard, such as some theoretical/philosophical problems or the NP-complete problems.

What if we try to compute the complexity not of the laws of physics, but of a given observer-moment/universe-state, and penalize the higher-complexity ones?

In chaotic systems, this actually works out to the speed prior: i. e., to assuming that the later steps of a program have less realityfluid than the early ones. Two lines of reasoning:

• The farther in time a state is,^[2] the more precisely you have to specify the initial conditions in order to hit it.
  • Justification: Suppose the program's initial state is parametrized by real numbers. As it evolves, ever-more-distant decimal digits become relevant. This means that, if you want to simulate the universe on a non-analog computer (i. e., a computer that doesn't use unlimited-precision reals) from $t=0$ to $t=n$ starting from some initial state $S_0$, with the simulation error never exceeding some value, the precision with which you have to specify $S_0$ scales with $n$. Indeed, as $n$ goes to infinity, so does the needed precision (i. e., the description length).
• Aside from picking the initial state that generates the observation, you also have to pinpoint that observation in the execution trace of the program. It can be as easy as defining the time-step (if you're working with classical mechanics), or as difficult as pointing at a specific Everett branch. And pinpointing generally gets more expensive with time (even in the trivial case of "pick a time-step", the length of the number you have to provide grows).

Anthropically, this means that the computations implementing us are (relatively) stable, and produce "interesting" states (relatively) quickly/in few steps.

• Suppose the microstate of a system is defined by a (set of) infinite-precision real numbers, corresponding to e. g. its coordinates in phase space.
• We define the coarse-graining as a truncation of those real numbers: i. e., we fix some degree of precision.
  • That degree of precision could be, for example, the Planck length.
• At the microstate level, the laws of physics may be deterministic and reversible.
• At the macrostate level, the laws of physics are stochastic and irreversible. We define them as a Markov process, with transition probabilities $P(x,y)$ defined as "the fraction of the microstates in the macrostate $x$ that map to the macrostate $y$ in the next moment".
• Over time, our ability to predict what state the system is in from our knowledge of its initial coarse-grained state + the laws of physics degrades.
  • Macroscopically, it's because of the properties of the specific stochastic dynamic we have to use (this is what most of the paper is proving, I think).
  • Microscopically, it's because ever-more-distant decimal digits in the definition of the initial state start influencing dynamics ever stronger. (See the multibaker map in Appendix A, the idea of "microscopic mixing" in a footnote, and also apparently Kolmogorov-Sinai entropy.)
• That is: in order to better pinpoint farther-in-time states, we would have to spend more bits (either by defining more fine-grained macrostates, or maybe by locating them in the execution trace).
• Thus: stochasticity, and the second law, are downstream of the fact that we cannot define the initial state with infinite precision.

So, Kolmogorov complexity is upper-semicomputable. This means that, for some $x$:

• You can prove an upper bound on $K\left(x\right)$, just by finding a program that computes $x$.
• You can only prove a lower bound on $K\left(x\right)$ using a program $p$ with $K\left(p\right)>K\left(x\right)$. Meaning, you can't use any fixed-size program (or formal system) to prove arbitrarily high complexity.

Imagine if it were otherwise, if some $p$ much smaller than $K\left(x\right)$ could prove a lower bound on $K\left(x\right)$. Then you could use that $p$ to cheaply pinpoint $x$: by setting up a program that goes through programs in order, uses $p$ to estimate the lower bound on their $K\left(x\right)$, then outputs the first program whose complexity is above a threshold. Which would simultaneously functions as an upper bound on $K\left(x\right)$: since our small program was able to compute it, $K\left(x\right)$ can't be higher than $K\left(p\right)$.

Thus, in order for arbitrarily complex states/programs to exist, it must be impossible to prove that they are complex.

Why? Why does that have to be the case?

Intuitively, it's because "proving" complexity requires pointing at specific features of the state $x$ and explaining why exactly they are complex. That is, your formal language must be expressive enough to precisely talk about those features, in their full detail. If, however, you can get away with using some abstractions/generalizations to prove $x$'s complexity, that by definition decreases $x$'s complexity.