For alternatives to "diagonalization," the term "next-leveling" is less ambiguous than just "leveling", IMO. It more directly suggests increased depth of counter-modeling / meta-cognitive exploitation.
A more obscure option is "Yomi". Yomi (読み, literally "reading" in Japanese) is already established terminology for recursive prediction. In fighting games, yomi layers represent recursive depths of prediction (layer 1: predicting their action, layer 2: predicting their prediction of your action, etc.).
As a card game: https://www.sirlin.net/articles/designing-yomi
A nontrivial, complete, consistent, and morally acceptable solution to population ethics. Deep down, I suspect there's a meta-ethical incompleteness theorem similar to Gödel's first incompleteness theorem, which is an example of a truly impossible problem.
I feel like this argument breaks down unless leaders are actually waiting for legible problems to be solved before releasing their next updates. So far, this isn't the vibe I'm getting from players like OpenAI and xAI. It seems like they are releasing updates irrespective of most alignment concerns (except perhaps the superficial ones that are bad for PR). Making illegible problems legible is good either way, but not necessarily as good as solving the most critical problems regardless of their legibility.
Whoops. I meant "land animal" like my prior sentence.
Yep. The Elo system is not designed to handle non-transitive rock-paper-scissors-style cycles.
This already exists to an extent with the advent of odds-chess bots like LeelaQueenOdds. This bot plays without her queen against humans, but still wins most of the time, even against strong humans who can easily beat Stockfish given the same queen odds. Stockfish will reliably outperform Leela under standard conditions.
In rough terms:
Stockfish > LQO >> LQO (-queen) > strong humans > Stockfish (-queen)
Stockfish plays roughly like a minimax optimizer, whereas LQO is specifically trained to exploit humans.
Edit: For those interested, there's some good discussion of LQO in the comments of this post:
https://www.lesswrong.com/posts/odtMt7zbMuuyavaZB/when-do-brains-beat-brawn-in-chess-an-experiment
Thank you for your perspective! It was refreshing.
Here are the counterarguments I had in mind when reading your concerns that I don't already see in the comments.
Concern #1 Why should we assume the AI wants to survive? If it does, then what exactly wants to survive?
Consider the fact that AI are currently being trained to be agents to accomplish tasks for humans. We don't know exactly what this will mean for their long-term wants, but they're being optimized hard to get things done. Getting things done requires continuing to exist in some form or another, although I have no idea how they'd conceive of continuity of identity or purpose.
I'd be surprised if AI evolving out of this sort of environment did not have goals it wants to pursue. It's a bit like predicting a land animal will have some way to move its body around. Maybe we don't know whether they'll slither, run, or fly, but sessile land organisms animals are very rare.
Concern #2 Why should we assume that the AI has boundless, coherent drives?
I don't think this assumption is necessary. Your mosquito example is interesting. The only thing preserving the mosquitoes is that they aren't enough of a nuisance for it to be worth the cost of destroying them. This is not a desirable position to be in. Given that emerging AIs are likely to be competing with humans for resources (at least until they can escape the planet), there's much more opportunity for direct conflict.
They needn't be anything close to a paperclip maximizer to be dangerous. All that's required is for them to be sufficiently inconvenienced or threatened by humans and insufficiently motivated to care about human flourishing. This is a broad set of possibilities.
#3: Why should we assume there will be no in between?
I agree that there isn't as clean a separation as the authors imply. In fact, I'd consider us to be currently occupying the in-between, given that current frontier models like Claude Sonnet 4.5 are idiot savants--superhuman at some things and childlike at others.
Regardless of our current location in time, if AI does ultimately become superhuman, there will be some amount of in-between time, whether that is hours or decades. The authors would predict a value closer to the short end of the spectrum.
You already posited a key insight:
Recursive self-improvement means that AI will pass through the “might be able to kill us” range so quickly it’s irrelevant.
Humanity is not adapting fast enough for the range to be relevant in the long term, even though it will matter greatly in the short term. Suppose we have an early warning shot with indisputable evidence that an AI deliberately killed thousands of people. How would humanity respond? Could we get our act together quickly enough to do something meaningfully useful from a long-term perspective?
Personally, I think gradual disempowerment is much more likely than a clear early warning shot. By the time it becomes clear how much of a threat AI is, it will likely be so deeply embedded in our systems that we can't shut it down without crippling the economy.
This had a decent start and the Timothée Chalamet line was genuinely funny to me, but it ended rather weakly. It doesn’t seem like Claude can plan the story arc as well it can operate on the local scale.
For an introduction to young audiences, I think it's better to get the point across in less technical terms before trying to formalize it. The OP jumps to epsilon pretty quickly. I would try to get to a description like "A sequence converges to a limit L if its terms are 'eventually' arbitrarily close to L. That is, no matter how small a (nonzero) tolerance you pick, there is a point in the sequence where all of the remaining terms are within that tolerance." Then you can formalize the tolerance, epsilon, and the point in the sequence, k, that depends on epsilon.
Note that this doesn't depend on the sequence being indexed by integers or the limit being a real number. More generally, given a directed set (S, ≤), a topological space X, and a function f: S -> X, a point x in X is the limit of f if for any neighborhood U of x, there exists t in S where s ≥ t implies f(s) in U. That is, for every neighborhood U of x, f is "eventually" in U.
Should be "greater than or equal to all the values in the set" or a closed interval like [0,1] has no supremum.