512

LESSWRONG
LW

511
Dialogue (format)EconomicsFictionAIRationality
Frontpage

85

The Tale of the Top-Tier Intellect

by Eliezer Yudkowsky
3rd Nov 2025
42 min read
53

85

Dialogue (format)EconomicsFictionAIRationality
Frontpage

85

The Tale of the Top-Tier Intellect
117Raemon
40Raemon
13khafra
14Raemon
11FireStormOOO
2MinusGix
14Raemon
7romeostevensit
4Raemon
6MichaelDickens
7Raemon
3plex
2Raemon
4plex
2Raemon
2Zach Stein-Perlman
3Mo Putera
49Simon Lermen
5A1987dM
13Simon Lermen
11Kenoubi
45cousin_it
5Eliezer Yudkowsky
63cousin_it
2[comment deleted]
34J Bostock
23romeostevensit
1Alex Vermillion
28Zack_M_Davis
22Cole Wyeth
2green_leaf
13A1987dM
2A1987dM
9Zack_M_Davis
12kbear
11AprilSR
5Adam Scherlis
6Jiro
4Warty
7ACCount
7Phaedrus
7CronoDAS
6Vermillion
2Alex Vermillion
5StartAtTheEnd
2Luiza
1Luiza
2joec
8CronoDAS
7Haiku
1MieszkoP
1StanislavKrym
1Bridgett Kay
-24Warty
New Comment
53 comments, sorted by
top scoring
Click to highlight new comments since: Today at 7:46 AM
[-]Raemon14d*11784

Serious question: (well, it'll start as "more of a comment, really", but, at the end I do have a question)

The comment: I think the world is bottlenecked on people understanding the sort of concepts in posts like these. I don't think the world is particularly bottlenecked on current-gen-Yudkowsky-shaped dialogue essays about it. They appeal to a small set of people. 

My guess is you write them anyway because they are pretty easy to write in your default style and maybe just mostly fun-for-their-own-sake. And when you're in higher-effort modes, you do also write things like If Anyone Builds It, that are shaped pretty different. And, probably these essays still help some people, and maybe they help workshop new analogies that eventually can be refined into If Anyone style books or podcast interviews.

But, that said, my questions are:

  • How much have you experimented with finding low-energy-but-different formats, that might reroll on who finds them compelling?
    • (I'm particularly interested in if there turns out to be anything short in this reference class)
  • How much have you (or, anyone else, this doesn't have to be you) systematically thought about how to improve the distribution channel of this sort of essay so it reaches more people?

Both of these are presumably high effort. I'm not sure if the the first one is a better use of your high-effort time than other things, or how likely it is to work out. But, wondering if this is an area you think you've already checked for low or mid-hanging fruit it.

(Also, having now thought about it for 5 min, I think this sort of thing would actually make a good youtube video that the Rational Animations people might do. That could be mostly outsourced)

Reply65
[-]Raemon14d4036

fwiw I did enjoy reading the first ~half of this (up through the last conversation with Mr Asi, and the beginning of the second conversation with Tessa). But after we started getting into each individual fallacy, I was like "okay I get it" and couldn't bring myself to keep reading. (I skipped to the end, which was a fine end, given the genre).

When I write regular blogposts, I often try to fit in examples that cover every major type of objection or concern I anticipate, but my impression is that people have the patience for more like 3 examples (rather than the 6-10 I end up writing), and just skim the rest, and it's sort of a fabricated option to cover everything.

Reply
[-]khafra14d1311

I feel like this was a sort of fractal parable, where the first two paragraphs should be enough to convey the point; but for readers who don't get it by then, it keeps beating you over the head with successively longer, more detailed, and more blatant forms of the point until the final denouement skips the "parable" part altogether.

Reply
[-]Raemon14d1412

Okay I think this would actually be kinda neat if there was a bit of UI that kinda explicitly says "if you get it, you can be done now", and if not you can keep reading.

Like, a "choose your own adventure" except the choices are "Cool, yeah I get Mr Humman's problem" and "What? Why do I care about this?". (Some more artistry would need to go into it, idk, but, feels kinda promising)

Reply
[-]FireStormOOO9d1113

This post struck me as venting as much as attempting to convince.  It really does capture the exasperation of needing someone to understand something that they've got their entire self worth wrapped up in avoiding.

Reply
[-]MinusGix14d2-3

Contrary, I liked this post and the latter half the most. It serves as a relatively direct parable about different levels of ability and also the major problems with common arguments against AGI/ASI, which I think people still miss making a point of very often. Spelling them out explicitly without going into super-long detail as a full post is good as it provides more concise argumentative handles. That is, people do not actually make the basic counterarguments enough.

(I also think those suggesting that this is already argued out enough should link to alternative posts. Posts for higher quality and more concise argumentation, and also posts made for reading by interlocutors.)

Reply
[-]Raemon14d149

(A question that should have been on the first list was "have you already actually checked that these sorts of posts have had the sort of outcome you want them to have?". Cuz, like, maybe they're totally working and are just kinda annoying for the people who already got the memo)

Reply
[-]romeostevensit14d74

Feels like more shades of error vs conflict theory. It doesn't matter if the ceos of ai companies are making mistakes if they are selected for being the sort of person who refuses to evaluate certain shapes of argument.

Reply
[-]Raemon14d42

Nod, though I expect there's value in various flavors of politician / public figures understanding these concepts who can put pressure on the CEOs.

(I dunno that that's the top theory-of-change I'd be spending my effort on, but, if my assumption is right that this is more like "getting marginally more utils out of low-effort-side-project-time", that's not exactly a crux)

Reply
[-]MichaelDickens14d62

maybe they help workshop new analogies that eventually can be refined into If Anyone style books or podcast interviews.

I think it's helpful to write arguments multiple times. And I think it's sensible to write out the argument in a "you-shape" and then refine it and try to make it more appealing to a broader range of people.

This kind of post might also give fuel for someone else to make basically the same argument, but in a different style, which could end up being helpful.

Reply
[-]Raemon14d70

Yeah I think that's my main causal story about it atm, I do think some previous pseudo-rants ended up in a more polished form in IABIED.

Reply
[-]plex14d30

I'd be keen to have good distillations of the yud things like this. It's kinda amusing how humanity's best explanations of several crucial concepts are dialogues like this. Maybe a nice first step is just collecting a list? My top one has been the logistic success curve for a while, must have asked like 5 writer for a distillation at this point.

Reply
[-]Raemon14d20

I'm not sure which logistics curve one you mean?

Reply
[-]plex14d42

Security Mindset and the Logistic Success Curve

…look, at some point in life we have to try to triage our efforts and give up on what can’t be salvaged. There’s often a logistic curve for success probabilities, you know? The distances are measured in multiplicative odds, not additive percentage points. You can’t take a project like this and assume that by putting in some more hard work, you can increase the absolute chance of success by 10%. More like, the odds of this project’s failure versus success start out as 1,000,000:1, and if we’re very polite and navigate around Mr. Topaz’s sense that he is higher-status than us and manage to explain a few tips to him without ever sounding like we think we know something he doesn’t, we can quintuple his chances of success and send the odds to 200,000:1. Which is to say that in the world of percentage points, the odds go from 0.0% to 0.0%. That’s one way to look at the “law of continued failure”.

If you had the kind of project where the fundamentals implied, say, a 15% chance of success, you’d then be on the right part of the logistic curve, and in that case it could make a lot of sense to hunt for ways to bump that up to a 30% or 80% chance.

 

Capturing the point that with a strong inside view, it's not unreasonable to have probabilities which look extreme to someone who's relying on outside view and fuzzy stuff. Strong Evidence is Common gets some of it, but there's no nicely linkable doc where you can point someone who says "woah, you have >95%/99/99.99 p(doom)? that's unjustifiable!"

Ideally the post would also capture the thing about how exchanging updates about the world by swapping gears is vastly more productive than swapping/averaging conclusion-probabilities, so speaking from the world as you see it rather than the mixture of other people's black box guesses you expect to win prediction markets is the epistemicaly virtuous move.

Reply
[-]Raemon14d20

Thanks to both you and Zack Stein-Perlman. 

One thing I immediately note is "wow the Logistics Success Curve jargon is particularly impenetrable", I think this could use a normie-friendly name.

Reply11
[-]Zach Stein-Perlman14d20

There's often a logistic curve for success probabilities, you know? The distances are measured in multiplicative odds, not additive percentage points. You can't take a project like this and assume that by putting in some more hard work, you can increase the absolute chance of success by 10%. More like, the odds of this project's failure versus success start out as 1,000,000:1, and if we're very polite and navigate around Mr. Topaz's sense that he is higher-status than us and manage to explain a few tips to him without ever sounding like we think we know something he doesn't, we can quintuple his chances of success and send the odds to 200,000:1. Which is to say that in the world of percentage points, the odds go from 0.0% to 0.0%. That's one way to look at the “law of continued failure”.

If you had the kind of project where the fundamentals implied, say, a 15% chance of success, you’d then be on the right part of the logistic curve, and in that case it could make a lot of sense to hunt for ways to bump that up to a 30% or 80% chance.

Reply2
[-]Mo Putera14d30

My guess is you write them anyway because they are pretty easy to write in your default style and maybe just mostly fun-for-their-own-sake.

This was essentially what he told Dwarkesh:

Dwarkesh Patel 3:43:54

Okay, so this feels like a good place to pause the AI conversation, and there’s many other things to ask you about given your decades of writing and millions of words. I think what some people might not know is the millions and millions and millions of words of science fiction and fan fiction that you’ve written. I want to understand when, in your view, is it better to explain something through fiction than nonfiction?

Eliezer Yudkowsky 3:44:17

When you’re trying to convey experience rather than knowledge, or when it’s just much easier to write fiction and you can produce 100,000 words of fiction with the same effort it would take you to produce 10,000 words of nonfiction? Those are both pretty good reasons.

Dwarkesh Patel 3:44:30

On the second point, it seems like when you’re writing this fiction, not only are you covering the same heady topics that you include in your nonfiction, but there’s also the added complication of plot and characters. It’s surprising to me that that’s easier than just verbalizing the sort of the topics themselves.

Eliezer Yudkowsky 3:44:51

Well, partially because it’s more fun. That is an actual factor, ain’t going to lie. And sometimes it’s something like, a bunch of what you get in the fiction is just the lecture that the character would deliver in that situation, the thoughts the character would have in that situation. There’s only one piece of fiction of mine where there’s literally a character giving lectures because he arrived on another planet and now has to lecture about science to them. That one is Project lawful. You know about Project Lawful?

Dwarkesh Patel 3:45:28

I know about it. I have not read it yet.

Eliezer Yudkowsky 3:45:30

Most of my fiction is not about somebody arriving on another planet who has to deliver lectures. There I was being a bit deliberately like, — “Yeah, I’m going to just do it with Project Lawful. I’m going to just do it. They say nobody should ever do it, and I don’t care. I’m doing it ever ways. I’m going to have my character actually launch into the lectures.” The lectures aren’t really the parts I’m proud about. It’s like where you have the life or death, deathnote style battle of wits that is centering around a series of Bayesian updates and making that actually work because it’s where I’m like — “Yeah, I think I actually pulled that off. And I’m not sure a single other writer on the face of this planet could have made that work as a plot device.” But that said, the nonfiction is like, I’m explaining this thing, I’m explaining the prerequisite, I’m explaining the prerequisites to the prerequisites. And then in fiction, it’s more just, well, this character happens to think of this thing and the character happens to think of that thing, but you got to actually see the character using it. So it’s less organized. It’s less organized as knowledge. And that’s why it’s easier to write.

Reply
[-]Simon Lermen14d499

As you can see, the serious adults in the room at DeepMind are really using comparative advantage as a reason AI will be fine:

https://www.lesswrong.com/posts/tBr4AtpPmwhgfG4Mw/comparative-advantage-and-ai

Reply
[-]A1987dM14d5-2

[ETA: Without following the link,] I have no idea whether you agree with them and "serious adults" is supposed to imply Eliezer isn't one, or you agree with Eliezer and mean "serious adults" sarcastically

Reply2211
[-]Simon Lermen14d130

I agree with Eliezer. I am trying to express that despite this story being seemingly a caricature of a silly person, this is close to how real experts at the doom labs are thinking. I read the story and thought it was good though I can see where some of the criticism in the comments may come from. My guess is that the story is trying to express: "don't overestimate people, don't be surprised in the future if humanity does really dumb stuff. Calibrate yourself that people believe really dumb arguments." But not sure, maybe he was just venting.

Reply
[-]Kenoubi14d116

"As you can see", "serious adults", "really" and "fine" all (mildly) demonstrate a sense of incredulity. "Look what these people actually believe! Just in case you thought it was a strawman." It's admittedly subtle, not stated, and I can see how someone could miss it. (I'll feel pretty stupid if I'm wrong.)

Reply
[-]cousin_it14d4546

Reading this felt like watching someone kick a dead horse for 30 straight minutes, except at the 21st minute the guy forgets for a second that he needs to kick the horse, turns to the camera and makes a couple really good jokes. (The bit where they try and fail to change the topic reminded me of the "who reads this stuff" bit in HPMOR, one of the finest bits you ever wrote in my opinion.) Then the guy remembers himself, resumes kicking the horse and it continues in that manner until the end.

By which I'm trying to say, though not in a top-tier literary way maybe, that you're a cool writer. A cool writer who has convinced himself that he has to be a horse-kicker, otherwise the world will end. And I do agree that the world will end! But... hmm how to put it... there is maybe a more optimal ratio of cool writing to horse-kicking, which HPMOR often achieved. Which made it more effective at saving the world, more fun to read, and maybe more fun to write as well.

Though I could be wrong about that. Maybe the cool bit in the middle wasn't a release valve for you, but actually took more effort than laying out the arguments in the rest of the essay. In that case never mind.

Reply2
[-]Eliezer Yudkowsky14d53

See Simon Lerner above on how dead the horse appears to be.

Reply
[-]cousin_it14d6362

It seems you interpreted my comment as "the essay argues against something nobody believes anyway". What I meant was more like "the essay keeps making its point in an angry and tedious way, over and over".

Reply
[+][comment deleted]14d20
[-]J Bostock14d3441

It's getting difficult for me to read these, and I basically agree with you on everything already and have read basically everything you've ever written (except Project Lawful). I don't think that anyone who disagrees with you will come away from this with a positive impression of your views or you as a person. 

This story overstays its welcome by not being particularly clever or funny. The stand-in for your opponents is so comically unlikeable that it reads like a caricature; my brain assumes that Humman is a strawman even though he (apparently) isn't because of the utter disdain that drips from the way you've written him. This might have worked in short form but this is a "42 minute read" by LW's metrics.

Reply3
[-]romeostevensit14d2311

If you try to have conversations about things that actually matter, many humans immediately become exactly that unlikeable. It's through social conditioning that we mostly learn to stop talking about things that matter because it goes so poorly.

Reply41
[-]Alex Vermillion12d17

I don't think this is true. This sounds like an issue that stems from the manner in which one approaches conversations, not the sorts of things one talks about.

I do not expect a longer comment on discussion to be useful to this thread (or, more importantly, to the post it's under), but I would like to put some chips down on the idea that talking to people in respectful and non-smug ways can be a good way to talk about "things that actually matter".

Reply
[-]Zack_M_Davis14d2816

This story would have benefited from being edited by a chess player. I think one of the better players in even a "medium-small town" with "a thriving chess club as one of its central civic institutions" would know more about the game than the author seems to. (The chess writing seemed off to me, and I am significantly worse than a serious club player.)

"I thought at first it was a mistake, for you to castle so early" is a weird thing for Humman to say. Castling early is standard default beginner advice. Even if there was some unusual feature of the opening that made it a bad choice in this game, you wouldn't use the word "so" in that sentence.

It's weird for Assi to describe Humman's play as using "particular tactics", and then to (insincerely) compliment him for "doing well at one-move lookahead" and not "unforcedly throwing away material right on your next move". Tactics are short sequences of moves that work together to achieve a goal. (An example I keep falling for in bullet games with the Englund gambit accepted (1. d4 e5?! 2. dxe5) opening: Black's dark-square bishop is on d6, White's queen is still on d1, the d-file is open due to accepting the Englund gambit, and Black castles queenside to put a rook on d8. Black sacrifices the Bishop with Bh2+, revealing a discovered attack of the Black rook on the White queen, which White can't do anything about because they have to use their move to deal with the check.) If a player is at the level of using "particular tactics", an IM who wants to complement them for social reasons shouldn't find it difficult (to the point of giving up after "a dozen seconds of" "twist[ing] his brain around") to find something concrete and nice to say that's less patronizing than "at least you're not hanging pieces."

(Also, the Ethiopean isn't a real opening; a cutsey fake detail like that feels out of place mixed in with real details like IMs needing an Elo of 2400, and I'd expect a club player to have heard of simuls.)

Do these flaws matter, given that the story isn't really about chess? I argue that it does matter, because a story that is about the folly of misperceiving how high skill ladders go should take basic care to get the details right concerning the skill ladder of its notional real-world example. (An earlier draft of this comment continued, "particularly in 2025 when basic care is so cheap. In the story, Tessa has no qualms about using LLMs to fill in domain knowledge gaps; why doesn't Yudkowsky?", but when I checked, Claude Sonnet 4.5 didn't anticipate my criticism.)

Reply
[-]Cole Wyeth14d2222

I don’t understand the target audience for this post.

I generally like Eliezer’s writing. However, I expect that most rationalists will find this post redundant, and even a bit patronizing - obviously this is a metaphor for humans vs ASI, but we have to read basically the same arguments and counterarguments all over again except explicitly about ASI the second time. If anyone couldn’t tell it was about ASI, I doubt they would have made it that far (because it would just seem like a very weird and implausible story). 

I felt the same way about the last post from Eliezer. I guess these just aren’t intended for me, because I’m not getting much out of them (and at least they could be compressed by five times). But I don’t really understand who they are intended for. I certainly wouldn’t send these to anyone in a discussion about ASI X-risk, because that would actually just be kind of rude. 

Also the ending is bad. The ending of the last one was also bad. 

Reply
[-]green_leaf14d20

I guess these just aren’t intended for me, because I’m not getting much out of them

Perhaps you already knew the underlying concept?

Reply
[-]A1987dM14d137

The main issue with this is that the only people who will actually finish reading this are the ones who already agree with your points (or, at least, disagree with them for reasons other than not being familiar with them), and hence don't need to read this (except for the sake of reading fiction as leisure).  Everybody else is going to stop reading a few paragraphs in.

Reply
[-]A1987dM14d20

(as for me, I read it because I woke up one hour before my alarm clock, spent half of that unsuccessfully trying to fall back asleep, didn't feel like getting up, and my phone was already on my nightstand)

Reply
[-]Zack_M_Davis14d9-4

Suppose we compare that whole function with Mr. Neumman's function, and compare how good are the probable moves you'd make versus him making. On most chess positions, Mr. Neumann's move would probably be better. [...] That's the detailed complicated actually-true underlying reality that explains why the Elo system works to make excellent predictions about who beats who at chess.

This explanation is bogus. (Obviously, the conclusion that Elo scores are practically meaningful is correct, but that's not an excuse.)

Mr. Humman could locally-validly reply that Tessa is begging the question by assuming that there's a fact of the matter as to one move being "better" than another in a position. Whether a move is "good" depends on what the opponent does. Why can't there be a rock-paper-scissors–like structure, where in some position, 12. ...Ne4 is good against positional players and bad against tactical players?

Earlier, Tessa does appeal to player comparisons being "mostly transitive most of the time"—but only as something that "didn't have to be true in real life", which seems to contradict the claim that some moves in a position are better on the objective merits of the position, rather than merely with respect to the tendencies of some given population of players.

The actual detailed complicated actually-true underlying reality is that by virtue of being a finite zero-sum game, chess fulfills the conditions of the minimax theorem, which implies that there exists an inexploitable strategy. You can have rock-paper-scissors–like cycles among particular strategies, but the minimax strategy does no worse than any of them.

The implications for real-world non-perfect play are subtler. As a start, Czarnecki et al. 2020 (of Deepmind) suggest that "Real World Games Look Like Spinning Tops": there's a transitive "skill" dimension along which higher-skilled strategies beat lower-skilled ones, but at any given skill level, there's a non-transitive rock-paper-scissors–like plethora of strategies, which explains how players of equal skill can nevertheless have distinctive styles. The size of the non-transitive dimension thins out as skill increases (away from the "base" of the top—see the figures in the paper).

This picture seems to suggest that rather than being total nonsense, the problem with Humman's worldview is in his attribution of it to the "top tier". Non-transitivity is real and significant in human life—but gradually less so as we approach the limit of optimality.

Reply1
[-]kbear14d120

This picture seems to suggest that rather than being total nonsense, the problem with Humman's worldview is in his attribution of it to the "top tier". Non-transitivity is real and significant in human life—but gradually less so as we approach the limit of optimality.

i was with you until this, but i must pick a nit. Humman's error is far earlier: he is not near that limit.

even in the "spinning top" model, we would expect that non-transitivity rarely matters: between two players picked at random, skill differences will dominate. you'll need strong selection effects (players selected from a particular Elo bracket, e.g.) before play-style can matter.

perhaps paradoxically, i would expect more non-transitive matchups at the very highest level, as that is where these selection effects are strongest. indeed, there are different "best" players for classical, blitz, 960, ... . at the local club, Neumman will win all of these, and the visiting IM will never lose a game, no matter the time control, or opening position.

Reply1
[-]AprilSR14d*115

I agree Tessa's explanation isn't especially good, though it's maybe more "incomplete" than "bogus".

I don't think the minimax theorem comes anywhere close to implying the existence of some sort of true optimal strategy, though, which I think becomes clear if you consider two types of chess bots. Bot A plays the same move as (something like) LeelaPieceOdds—unless that move would be moving from a game theoretically won position to a draw or loss, or from a draw to a loss, in which case it, say, randomly selects from all the moves that don't do that, or ideally picks a move that humans are inclined to blunder against (maybe LeelaPieceOdds's second choice, or something.)

On the other hand, Bot B immediately resigns whenever the position is a game theoretic loss, and immediately offers a draw whenever the position is a game theoretic draw. If its opponent rejects the draw offer, Bot B prefers to stay in states with the fewest opportunities for its opponent to blunder.

While both are "inexploitable", Bot A beats humans every time, and Bot B draws them every time. (Unless chess is a game theoretic win for (WLOG) white, in which case Bot B wins as white but immediately resigns as black.) If you made Bot B a chess.com account, it very well might literally never break 1000 Elo.

So in pathological cases the non-transitivity can get pretty bad. The tops result sounds really neat but I haven't read it yet and don't know exactly how Bot B would fit into it—obviously "<1000 Elo" isn't really going to be a description of Bot B that captures how it fits into such a structure very well.

Reply1
[-]Adam Scherlis10d50

I like this observation!

To make the pair of strategies slightly more concrete, we could say that Bot A always picks LeelaPieceOdds's top choice, out of moves that don't hurt the game-theoretic status, and Bot B always picks its bottom choice out of those moves.

It would be fun to watch Bot B play a novice human, followed by a grandmaster. In both cases it might look a lot like a bored genius playing a newbie -- hanging some pieces to even the odds, then playing competently with what's left.

Reply1
[-]Jiro13d60

Why can’t there be a rock-paper-scissors–like structure, where in some position, 12. …Ne4 is good against positional players and bad against tactical players?

I would say that in that situation, the move is bad, but being a positional player after your opponent makes that move is also bad.

Reply
[-]Warty14d40

hmm it didn't strike me when reading but that paragraph in OP is kinda bad as you say.

> the minimax theorem

a less complicated and more correct reference is Zermelo's theorem, this is combinatorial game theory not economic (a further complication of reality is that the wiki article is kinda bad, it's just induction). The theory also explains that in fact some chess moves are better than others in a mathematical sense, because some worsen the position more than others (e.g. Win->Draw vs Win->Win). Though it doesn't match what Tessa says about vectors with lengths very well.

Reply11
[-]ACCount13d75

This one is bad. "I struggled to find a reason to keep reading it" level of bad. And that was despite me agreeing with basically every part of the message.

Far too preachy and condescending to be either convincing or entertaining. Takes too long to get to the points, makes too many points, and isn't at all elegant in conveying them. The vibe I get from this: an exhausted rant, sloppily dressed up into something resembling a story.

It's hard for me to imagine this piece actually changing minds, which seems to be the likely goal of writing it? But if that's the intent, then the execution falls short. Every point made here was already made elsewhere - more eloquent and more convincingly - including in other essays by the same author.

Reply
[-]Phaedrus13d73

I think essays like this are not very helpful for the AI safety agenda. In fact, they seem quite likely to do more harm than good?

I see dozens of arguments for and against specific AI risk models every day. A large fraction of these arguments (especially in the Twitter and podcast circles where people think deeply about AI and often work in AI risk or frontier labs) are against the Yudkowsky positions in IABIED. These are often arguments made by very smart people, well versed in the literature (including Yudkowsky's writing), who have significant meta-cognitive awareness when it comes to their own intellectual weaknesses.

All this is to say: of all these "live" debates that I see play out every day, none involve the type of IQ/intelligence skepticism that is criticized in this argument. People debate the effects on power-seeking of differing RL task lengths, the speed of super-exponential AI-assisted research takeoff, and the emergence of AI situational awareness with regard to testing environments in frontier models, but no one that I can see, in any significant position of power, is falling for the farcical reasoning employed by Mr. Humman. Even Simon Lermen's example of "misuse" of comparative advantage (https://www.lesswrong.com/posts/tBr4AtpPmwhgfG4Mw/comparative-advantage-and-ai) is a fine educational example for the specific labor dynamics of the period we're entering now, where AI use and human labor coexist. In any event other than the (unlikely) event of a ASI takeoff in hours, there will be a period of years if not decades where comparative advantage holds for human and AI labor. Is there anyone who's arguing from Ricardo about the likelihood of AI risk? Maybe Marc Andresseen types? IQ is even more lopsided — who in silicon valley in the year of our lord 2025 is an IQ denier? Is the Trump admin somehow unfamiliar with the concept that people or machines can be very smart?

Anyways, all this is to say: who is this post for? Why write and promote arguments against irrelevant incorrect positions instead of addressing the massive substantive issues where intelligent, informed people disagree? Why take a continual "outside view" of the AI risk debate instead of grappling with the substantive issues that sway the opinions of informed insiders at labs and in politics? Will smart critics of AI safety arguments resent that their side is represented in Yudkowsky posts by a cretin with the reflective acumen of a kindergartener?

Reply1
[-]CronoDAS14d71

How much more delusional would someone like Mr. Humman end up if, instead of chess, he had been playing a game with random elements and actually did beat the expert due to sheer luck?

Reply
[-]Vermillion13d60

Seeing lots of criticism is discouraging, so ill just say thanks Eliezer for writing it.

Reply
[-]Alex Vermillion12d20

Hello, name doppleganger. I also liked the story, though I'll admit to skimming the back half once it was unsurprising for a while (metaphors for something I already know about can be that way sometimes). I am imagining a short book of parables like this as an add-on to If Anyone Builds It to which a much shorter version of this (defending only on ONE front instead of MANY) would be a welcome addition.

Reply
[-]StartAtTheEnd14d52

Intelligence cannot be boiled down to a single number, but it can be boiled down to about five numbers. If you gave somebody who was unable to grasp high levels of abstraction inhuman processing speed (say, 15 standard deviations), then this qualitative difference would not make a quantitative difference (so FSIQ scores aren't equal, the subtests which result in the final score are important, too). Also, the intelligence distribution is a bit weird than our models suggest, which is why geniuses can be so far apart from eachother. Nikola Tesla could visualize his designs in 3D space, just how many bits of information could this visual working memory of his contain? The standard deviation of working memory items is roughly 1, so even 50 items 'should' only appear in one in 10^400 people.

I don't believe much in super-intelligence, but I have recently had the horrifying thought that most plans have a threshold on required intelligence, and that further intelligence doesn't make much of a difference (In short, that scaling is unreasonably effective). This means that an AI with 140~ IQ and 100000 times more actions per second than a human, could take over the world, even if I could beat it at some IQ test subsets. That competitive RTS game players aim to improve their APS (actions per second) speaks in favor of this idea.

Is there any reason you make Humman so obviously wrong and dislikable? If any reader believes something that Humman does, I think they will feel offended and close the page before they to read the counter-arguments to the beliefs he represents.

Ideally, by the way, we'd gatekeep things like AI research in a manner that people like Humman will be barred from entering. There are classes in topics like security and biology which are good at destroying many of the naive arguments Humman represents in the post.

Finally, beliefs like "everyone is born with the same amount of stat points" are spread even in Harvard (e.g. Howard Gardner's theory of multiple intelligences) and it's downstream from the social instincts which guard against outliers. The hard sciences are getting less hard by the year because of ideological, political and social dynamics, and these are directly responsible for the poor general understanding of intelligence. Human beings fail to understand AGI because they anthropomorphize it (they consider their own humanity axioms that all systems are bound by).

Reply
[-]Luiza13d21

Short of the redundancy and letdown ending, I like this writing as it captures echoes of reasoning failure (that I find myself fall for at times); seems to be written not just for AI researchers (& adjacents) in 2025, but for many minds across "now and then".

It strikes me that Humman does grasp truths (reality is complicated and people do have different strengths) but errs "true at this/his resolution" for "true at all scales".  Feels like he assumes self-similar scaling (like a tree) instead of considering the nature of realities that scale in non-self-similar ways (like a snowflake*, with shifts from dendritic to radial structures). More so, he uses his understanding of complexity as a thought-terminating invocation rather than a call to deeper/clearer/coherent-er modeling(s). Both are fairly common failure modes and it would be cool to leave them behind, but I'm unaware of stable ways to do so.

*not meaning to use the snowflake example as a supportive argument for his position ~ "every snowflake is unique and incomparable", tho I do like that in my best of days.

Reply
[-]Luiza12d10

Updated a bit on the self-similar vs non-self similar scaling; I'm more unsure than I previously thought that I understand how scaling works, from individual to different types of collectives + time dynamics. 

Reply
[-]joec14d22

I find it interesting how nobody seems to make Mr. Humman's arguments about chess, but plenty of people seem to make his arguments concerning ASI.

Now, if I were arguing with Mr. Humman, I'd get him to try to play against stockfish (or to play against me, and cheat using stockfish) to let him know what it really feels like to play against a superior mind. I feel like it would be pretty hard to continue arguing his point about chess after Stockfish punched his lights out for the 5th time in a row (or however long it takes to sink in). As a side-note, I do find it interesting that chess engines were only briefly mentioned here. Nor was a large lookup-table-type chess engine.

Now, it might be that there are no Mr. Hummans (with respect to chess) these days, but I wonder if there ever were any. That is, someone claiming that a machine would never be able to reliably beat a human at chess. Was this a notion that more than a handful of people actually held? Was it shattered by the advent of chess engines or something else?

Reply
[-]CronoDAS14d80

People used to make that argument about Go because its branching factor made AI programs that worked like chess engines intractable - until someone invented Monte Carlo tree search.

Reply
[-]Haiku13d71

I found this after a brief Wikipedia rabbit hole: an article from the 1982 North American Computer Chess Championship. https://www.cgwmuseum.org/galleries/issues/softline_1.3.pdf

On the evening of the last round, there was some discussion amongst tournament participants about when or whether a computer program might become chess champion of the world. Monroe Newborn, programmer of McGill University's Ostrich, predicted it could happen within five years. Valvo thought it would be more like ten, and the Spracklens were betting on fifteen years. Thompson thought it would be more than twenty years before a program could be written that would beat all comers, and a few others said it would never happen.

The most widely held view, however, was that a computer program would become world champion by or shortly after the year 2000. Considering both the complexity of the game and the complexity of the human mind, that seems like a remarkably positive outlook on the future of computing.

Garry Kasparov believed as late as 1989 that machines would never completely best humans in chess, and thought he personally would never be beaten by a machine. https://www.chesshistory.com/winter/extra/kasparovinterviews.html

Question: Two top grandmasters have gone down to chess computers: Portisch against “Leonardo” and Larsen against “Deep Thought”. It is well known that you have strong views on this subject. Will a computer be world champion, one day?

Kasparov: Ridiculous! A machine will always remain a machine, that is to say a tool to help the player work and prepare. Never shall I be beaten by a machine! Never will a program be invented which surpasses human intelligence. And when I say intelligence, I also mean intuition and imagination. Can you see a machine writing a novel or poetry? Better still, can you imagine a machine conducting this interview instead of you? With me replying to its questions?’

I was able to confirm that directly in the magazine here: https://escaleajeux.fr/?principal=/jeu/js_55?

Reply1
[-]MieszkoP3d10

I think the phenomenon of repression[1] has been captured well in this text. Mr. Humman knows that he is a weaker chess player than Mr. Neumann, and also knows that he is a weaker than Mr. Assi. For this reason, he avoids confronting the former and does not want to put any money on a slight loss against the latter.

I observe this phenomenon very often when I try to persuade people in IT to oppose the development of AI. It’s easier to deceive oneself than to confront an uncomfortable truth.

A lot of harm is done here by pop culture. Male–female relationships don’t look anything like they do in the Disney universe, work doesn’t look like it does in The Office, and young people don’t live the way they do in Friends. Good intentions aren’t enough to achieve success in life. Illness exists, death exists. We push these thoughts away. It’s similar when it comes to ASI.

  1. ^

    Repression is a psychological defense mechanism in which a person unconsciously blocks or pushes away thoughts, memories, or impulses that are too troubling, painful, or socially unacceptable to deal with.

    These unwanted mental contents don’t disappear — they are stored outside of conscious awareness and can still influence behavior, emotions, and decision-making. 

Reply
[-]StanislavKrym8d10

I think that one could also explain Elieser's idea in a different manner. DeepSeekV3[1] was trained on 14.8T tokens. Were it to experience 10-20 tokens/second, as humans likely do, it would require between 700B and 1.4T seconds. Since a year is built of ~31M seconds, the 700B seconds become ~23 thousands of years and the 1.4T seconds become ~46 thousands of years. 

A talented human at the peak of one's capabilities learns the skills necessary to contribute to an area or to excel in a competition like chess in at most 30 years since birth. An AI who lived experiences worth thousands, if not millions, of years will likely have encountered a major part of tricks that the human has to offer or even be able to outright predict the opponent's next most likely plan and the advisor's idea. Attempting to give advice to an ASI wouldn't be more useful than trying to teach someone who spent thousands of years studying all kinds of martial arts[2] to the point of instinctively predicting the opponent's next move to fight in a specific style.   

 

  1. ^

    Which I use because the precise number of its parameters and tokens on which it was pretrained have been made public. Closed-source models like o3 and o4-mini have only estimates like Kokotajlo's. 

  2. ^

    I strongly suspect that there exist Chinese web novels where the protagonist does exactly that, since there exists a Russian web novel where something similar happens.

Reply
[-]Bridgett Kay14d10

After all of Mr. Humman's condescension toward Ms. Tessa, I couldn't help but think of this when I read the ending. 

“The girl he had dragged along to the pub with him had grown to loathe him dearly over the last hour, and it would probably have been a great satisfaction to her to know that in a minute and a half or so he would suddenly evaporate into a whiff of hydrogen, ozone and carbon monoxide. However, when the moment came she would be too busy evaporating herself to notice it.”

Reply2
[+]Warty14d-2427
Moderation Log
More from Eliezer Yudkowsky
View more
Curated and popular this week
53Comments
Deleted by cousin_it, 11/04/2025
Reason: Comment deleted by its author.
Mentioned in
112Comparative advantage & AI

Once upon a time in the medium-small town of Skewers, Washington, there lived a 52-year-old man by the name of Mr. Humman, who considered himself a top-tier chess-player.  Now, Mr. Humman was not generally considered the strongest player in town; if you asked the other inhabitants of Skewers, most of them would've named Mr. Neumann as their town's chess champion.  But Mr. Humman did not see things that way himself.  On Humman's theory, he was really quite good at the Ethiopian opening and variation in chess, while Neumann was more of an all-rounder; a jack of all trades, and therefore, of logical necessity, master of none.  There were certain tiers of ability in the town chess club, and Humman and Neumann were both in the top tier, according to Mr. Humman, and that was all you could really say about it, according to Mr. Humman.

Humman did not often play against Neumann directly; they had not played in a few years, in fact.  If you asked Humman why not, he might have said that it was more gracious to give younger players the chance to play him, rather than the top-tier chess-players being too exclusive among themselves.  But in truth that was a sort of question Mr. Humman would not think about spontaneously or ask himself without outside prompting.  Humman was not the sort to go around comparing himself mentally to Neumann all the time.  Humman was satisfied to have reached the top tier of chess ability, without going around comparing himself to his fellow top-tier players.

One week it came to pass that a FIDE-rated International Master of chess was visiting their small town, to meet the family there of a woman that he was dating.  The visiting Master had been much chuffed to hear that the town of Skewers had a thriving chess club as one of its central civic institutions, and so the Master offered to play one game each against anyone interested, over the next few days.  Mr. Assi, was his name.

One of the less polite young ladies of the town, whom some might have called a troll, jokingly asked Humman at the town's grocery how he fancied his chances against Mr. Assi.

In truth Mr. Humman had not really chosen, as such, to play a game against Mr. Assi.  But everyone around him seemed to take so much for granted that he would, that there didn't seem to be any face-preserving way not to.  So Mr. Humman thought about the young lady's question a few moments, and said, "Forty-sixty; in his favor, that is."

The young lady didn't spit coffee all over herself and ruin her dress, but only because she wasn't drinking any coffee.  "Forty-sixty?" she said.  "...Oh.  You're joking.  Totally got me, too.  Well-played."

"Why would I be joking?" said Humman, sounding quite sincere.  (Mr. Humman was not in town famed for having a sense of humor.)

The woman stared at him a bit.  "Hold on," she said, and quickly murmured something into her cellphone, and read something off its screen.  Then she said, in the authoritative tones of somebody who had no doubt already known that answer all along and had only been double-checking it, "Mr. Humman, an International Master is someone with a FIDE-recognized Elo score of 2400 or higher.  To have 40-60 odds against 2400 Elo would require you to be ranked 2330.  You are not an Elo 2330 chessplayer, Mr. Humman."

"Oh, my dear young lady," Mr. Humman said, quite kindly as was his habit when talking to pretty women potentially inside his self-assessed strike zone, "a simple little number like that cannot possibly summarize the playing style of a top-tier chess-player like myself.  Some players are more adept with the forking tactic, others with the skewering tactic; others at building solid pawn-positions; others choose a particular opening or line of play and learn all its ins and outs.  There is not, inside a chess-player's brain, a little generic engine with a little generic number, one single little number that determines how strong is their whole style of play.  None of us are strictly better than any other -- at least, not among top-tier players like myself.  All of us are weaker in some places, and stronger in others; so nobody has the right to look down on anyone else, once they've reached the top tier of chess."

"Pffffffft," said the dear young lady.  "Did you have the sort of parents who claimed to you as a child that life was fair?  My own parents always told me the opposite, and while they were kind of jerks about it, that doesn't make them wrong."

"Once you've reached the top tier of chess yourself, you will understand how it goes," Mr. Humman continued to kindly explain to her.  "Inside any chess player is the summarized and distilled memories of all the games we've played, and all the lessons we've learned the hard way.  No two chess-players have the same set of games to remember, or have learned all the same hard-learned lessons.  As a professional player, who gets paid to play chess and need do nothing else, Mr. Assi may have played twice or even three times as many games as I have.  But that's not the same as him having three-to-one odds of winning against me!  Many of the most important chess lessons are among the first lessons learned, you see.  After that you cannot learn those lessons twice or thrice again, and have a similar jump in playing ability each time.  Once you know how to fork two pieces with a knight, and you've played out a few games like that, someone like Mr. Assi may only know how to do it 10% better than I do, after playing three times as many games."

"Want to bet money on whether you win against Mr. Assi?" said the young lady.  "Ideally, like, a lot of money."

"Why no, of course not," said Mr. Humman.  "Mr. Assi probably has played a few more games than me, for all that he is younger; and so it is more likely for me to lose, than to win.  Why would I bet if I expected to probably lose?  What a silly idea."

"I'll offer you three-to-one odds," said the young lady, "which, given your self-assessed 40% chance of winning, implies that the Kelly criterion says you should bet --"  She paused to whisper to her cellphone.  "20% Of your total bankroll, which in principle doesn't only include your bank account savings but also any expected future income.  Hundred bucks sound about right?"

"I'm a chessplayer, not a gambler," Mr. Humman sniffed, and went upon his way.

But in truth, the young lady had started Mr. Humman thinking; and even, thinking and rethinking about his life.


Soon enough then the appointed day came to pass, that Mr. Assi began playing some of the town's players, defeating them all without exception.  Mr. Assi did sometimes let some of the youngest children take a piece or two, of his, and get very excited about that, but he did not go so far as to let them win.  It wasn't even so much that Mr. Assi had his pride, although he did, but that he also had his honesty; Mr. Assi would have felt bad about deceiving anyone in that way, even a child, almost as if children were people.

As for Mr. Humman, he did still have his day-job and so did not linger long about the chess center while Mr. Assi was playing others.  The gossiped word did happen to find way to Mr. Humman, that Mr. Assi had not yet lost a single game, but Mr. Humman was not fazed by hearing that.  After all, few others in the town of Skewers had reached the top tier of chess-players like himself.  Even if Mr. Assi had happened to beat Mr. Neumann, what of that?  The odds of that outcome had already been 40-60.

So the appointed day turned into the appointed hour, and Mr. Humman sat across from Mr. Assi.

Mr. Humman had decided, after some strange internal twinges that had made his brain feel uncomfortable, not to play his strongest Ethiopian opening variation against Mr. Assi.  It wouldn't quite be sporting, after all, in a friendly little match like that, for Mr. Humman to play his most aggressive and experienced opening, which Mr. Assi could hardly be expected to have memorized.  If that made Mr. Humman's odds of winning a bit worse, what of it?  It was just a friendly little match after all.  There was no need for top-tier chess-players to compare themselves to one another, or try to show themselves better than each other, when none could be truly superior.

So the game began, and then continued.  On each turn Mr. Humman would peer long at the chessboard, and finally make a move; upon which Mr. Assi would glance up from his laptop, immediately counter-move, and then go back to editing some essay he was working on.  Indeed, Mr. Assi was playing two other players in Skewers simultaneously, to save time and make sure everyone got a chance.

Mr. Humman had to admit he found that part impressive.  Humman had not realized up until this point that, as a professional continued practicing, a pro could continue to gain in speed and the ability to play multiple games in parallel -- even if, logically, there could be only so many truths to learn about chess as such.  There was even a kind of visceral shock to it, to see Mr. Assi moving so fast; it came across visibly as something that Humman himself could not have done.  The man well deserved his title of International Master.

As for the ending of the game, it did happen that Mr. Humman lost -- as Humman had frankly expected and confessed would probably be the case -- all the more so, as Humman had not opted to play his most practiced Ethiopian variation.  Though, Mr. Humman felt, he had put up a good fight, there; it had not been clear (to Humman) that Mr. Assi was bound to win, until near the very end, when Mr. Assi had taken Mr. Humman's last remaining rook.  That sort of thing happened in chess, of course; Mr. Humman had had no way of foreseeing that the current line of play would end by giving Mr. Assi that opportunity.  Really, in a way the game had been settled by Luck.  Mr. Hamman was a firm believer in the doctrine that no chess-player could be beyond the vagaries of Luck.  But Mr. Assi had undeniably played with great competence up until then; one could hardly win by lucky opportunity, without having played well enough to get that far.

"That was some excellent play you put forth there," Mr. Humman said afterward to Mr. Assi; quite sincerely, for Humman believed in giving others all of the compliments that were their just due.  "I thought at first it was a mistake, for you to castle so early, and then break up your pawn wall, but you defended the resulting vulnerabilities very well."

Mr. Assi's eyes betrayed a look of some slight confusion, as if he was not sure what sort of conversation he had landed in.  But Mr. Assi's mouth said at once, as quickly as he'd replied to each chess move in the game, "Thank you very much."

"What did you think of my own game?" Mr. Humman inquired.

"Fundamentally, what you must develop at this point in your journey is foresight," said Mr. Assi, still without any delay in answering.  "You arrange positions that seem to you to be statically strong.  Your play alternates between trying to arrange static defenses, and trying particular tactics to assault me.  You lack a felt sense of where the board will be five moves, fifteen moves later.  I would guess you are not even trying much to imagine it.  You do not feel how your current static defense will later become vulnerable.  Instead you spend moves and initiative on particular tactics, while the larger game goes on around you.  I cannot read your mind, of course, but it is a plausible guess at diagnosing you, because that is a common place for players of your level to get stuck -- that only the current state of the board feels real to them.  So you must try to train your foresight, and that begins by at least attempting to make predictions about self-consistent ways the board could look later.  Your predictions will be all wrong, but that is how practice begins."

"Oh, well," Mr. Humman said, "I had really hoped more to hear of where you felt my own play was strong, or clever -- the same sort of perspective that I offered you."  Mr. Humman kept any felt offense out of his voice; Humman was aware that not everyone could be as adept as he himself was, at social graces.  Humman knew there was a sort of clueless person who could not help but reply to your compliments with criticism, if you didn't remind them otherwise; Humman had met such people many times.

To this, Mr. Assi did not reply immediately.  His mouth quirked, briefly, before being controlled to greater slackness.  His eyes went to the chessboard, as if to review mentally how the game had played out.  (For the most part, Assi had played defensively and only made good moves in response to assaults, rather than exploiting the many many flaws in Humman's own fortifications, so as not to end the game too quickly.  It was still possible that way for the opposing player to learn something, and they'd have more time to learn.)

It was in fact something of a challenge -- and Mr. Assi was not one to turn down challenges immediately, before even trying them, especially in the realm of chess -- to look through all of the disastrous play that Assi had tolerated, and try to twist his brain around to look for something that could be complimented instead.

After a dozen seconds of giving that a fair try, Mr. Assi decided that it was in fact too much work, and gave up.

Also a more social part of Mr. Assi's brain had completed something of a guess about the level of intellect that he was talking to, here, and the sort of vulnerability it might have to particular sequences of words.

"You are doing well at one-move lookahead, at considering all the immediate consequences of a chess move," said Mr. Assi.  "I don't recall any occasions where you made the sort of blunders that beginning players do, unforcedly throwing away material right on your next move.  That is not something that every player at this club could say."

Mr. Humman beamed back at Mr. Assi, feeling more secure now in their completed friendly exchange of compliments, and how it had broken the ice.  "I was wondering, in fact," said Mr. Humman, "if there might be a chance for me to become a professional chess player, myself.  I have felt a little cooped up, in our little town of Skewers, of late.  I was wondering if I ought to take my chess game on the road."

Mr. Assi did not immediately tell Mr. Humman no; for it was not Assi's way to immediately judge that other people ought not to dream, or should not try to grow beyond their present levels.  "Practice hard in the online arena," said Mr. Assi, "or against machine players, and see if you are making progress.  Machine ratings are excellent, these days; they will tell you accurately where you fall relative to the least professionals."

"Well, in truth," Mr. Humman said, smiling more widely, "I was wondering if I could start by being part of an expert duo, with you -- playing two-person chess games, together.  I do realize my play still has some weaknesses, but you could shore up my weaknesses; and I could shore up some of yours, I'm sure."

There was, then, a small, but perceptible, pause, on the part of Mr. Assi.  His eyes widened, and his mouth quirked again, before Mr. Assi brought himself under control.

"That is known as team consultation chess," Mr. Assi said, having planned a reply with what Assi himself considered to be really quite exceptional and praiseworthy tact.  "Alas, I'm afraid that it is a very small niche, which FIDE does not even bother to rate.  It's not where I am interested in going with my career.  So no, but thank you for the compliment of the offer."

"Well, could you introduce me to another professional who might be interested, then?" said Mr. Humman.  "It seems to me, logically, that pair consultation chess should not be a small niche, and maybe we can make it a bigger one.  Two heads must certainly be better than one, since it is realistically impossible for any two chess players to share the same set of experiences, and so we all develop different strengths and weaknesses.  Or a team of four top-tier players, say, if we don't stop at just two, ought to be able to crush any chessplayer at all."

"I see," said Mr. Assi in a tone of somewhat helpless fascination.  "So... starting from the admittedly true premise that every chess-player has different strengths and weaknesses... you conclude that you... yourself... ought to have strengths that could help cover... my weaknesses."

"Well, of course!" said Mr. Humman.  "Surely you're not implying there's nothing I could contribute to assist your play."

"I am certain there are a great many things you know that I do not," said Mr. Assi, "and much you could teach me, if I knew what questions to ask, for no two different experiences of life are the same, as you say.  But not in the realm of chess, to be frank.  For though the possibilities of chess are endless, they are not as wide as the Earth.  In the smaller universe of chess, it is possible for one player to be just better than another; and so it is unlikely there is any good advice you could knowingly offer me, in a serious game."

"I never!" exclaimed Mr. Humman in genuine shock.  "Do you think you can conclude you're just better than I am, on the basis of one game that you happened to win at the end?"

"Others wish to play me," said Mr. Assi, "and I am afraid that I must firmly request you to get up from this chair and yield your place at the table to them."


Later that evening Mr. Humman was shopping at their town's grocery, which was one of its civic institutions to a greater extent even than its chess club, when he again crossed paths with that young lady who was sometimes considered something of a troll.  ("Tessa" was the name she went by, these days, short for her online handle of Socratessa.)

Now the thing about the day's previous events, was that they had taken place in earshot of the other two players facing Mr. Assi in simultaneous chess.  If you had measured the speed at which the resulting gossip had propagated across Skewers, Washington -- measured it very carefully, and with sufficiently fine instrumentation -- it might have been found to travel faster than the speed of light in vacuum.  The gossip had been retold essentially correctly, even.  There had been two eyewitnesses, both of whom had made themselves available for questioning immediately after the event; and neither of the two had been the sort to lie out of sheer existential habit, when mere truth was delicious enough to serve uncooked.

It would be only natural then, and expected, for a troll to pounce on Mr. Humman with all the delighted eagerness of a shark scenting blood.

"Uh, hi," the woman said gingerly to Mr. Humman, when she saw him at the grocery.  "I heard you had a bad experience today.  I hope it didn't crush your soul too much -- or, uh, actually, I should say, uh, we don't have to talk about it if you don't wanna."

Tessa knew she would never be among the best of all good people, but she tried to be a good person nonetheless.

"I have never met a chess-player so egotistical in all my life!" stormed Mr. Humman.  "That Assi fellow thinks he has nothing left to learn, and is uninterested in any other person's assistance or even advice!  Every breath he breathed showed how he thought himself better than me, and he wasn't politely hiding that feeling like I do!  I seriously believe that man was so incredibly, insanely arrogant that he was holding to his own opinions without moving in the direction of mine at all!"

"Wow," said the woman.  Her hand crept down to her cellphone, considering whether to start recording what might be an incredibly popular social media video.  But then she thought better of it, and halted before she could condemn Humman to eternal notoriety as a meme.  She was being good.  She was being good.  She was at least not being too awful.  She was being good.  "Well, in that case Mr. Assi should have crushed your soul a little harder, because it sounds like you've gone past just being resilient to trauma, and into the realm of completely failing to learn from experience."

"Well, of course you'd think so," Mr. Humman said.  "You think every chess-player can be reduced to a featureless single number powering a little generic engine inside their heads; and if one player's imaginary number is greater than another's, that's the only thing that matters about either of them."

"The idea that every player contains a tiny generic engine powered by a single number is just not what an Elo score is, and it's not something that needs to be true for an Elo score to be useful," said Tessa.  After the embarrassment of needing to look things up in Gemini, she'd made sure to put more knowledge inside her own head for next time.  "More like, if player 1 beats player 2 most of the time, and player 2 beats player 3 most of the time, then probably player 1 will beat player 3 most of the time.  If the comparison between clearly unequal players is mostly transitive most of the time, that is sort of like players being laid out on a global line.  It didn't have to be true in real life, but it is true in real life, that when player 1 beats player 2, and player 2 beats player 3, you have learned something that is helpful for guessing a chance that player 1 beats player 3.  Their chance of beating each other, the quantitative probability, is like a kind of directional distance.  So from there, we can ask where people would be on a global line if there was a global line."

"No giant floating line like that actually exists in the real world," said Mr. Humman.  "We can ignore it the same way we ignore talk of ghosts and goblins, which also don't exist.  Why, just last week, Mr. Chimzee beat me at a chess game, even though usually I beat him.  Why?  Because I had slept poorly the previous night.  What can the theory of the Elo numbers floating above our heads, say to that?  I'll answer for you: it can say nothing.  It retires in shame from the field of scientific hypotheses, defeated and falsified.  In real life, one player is more adept with the tactic of forks, another player is more adept with the tactic of skewers, and their strengths vary by the day with how much sleep they've had.  Real reality is complicated -- though I understand that's hard to appreciate for young people like you, and only we old and wise people truly get it in our guts."

"If reality was complicated in a way that didn't mostly line up with Elo scores, the Elo scores wouldn't actually work to make predictions," said Tessa.  "When you sum up all your subskills plus all the extra factors like 'how much sleep you've had' and 'how much sleep Mr. Chimzee has had', it works out to you beating Mr. Chimzee 75% of the time, not to it being 50-50.  And then somebody who's played more chess than both of you and also was born with more talent, who learns faster from playing fewer games, is likely to be more adept with forks and more adept with skewers.  That's why Mr. Neumann can be, in general, a better chess player than you, and kept winning games against you until you refused to play him again; and stopped even thinking about his existence, I'd bet, judging by how you never talked about him again."

"Well, no," said Mr. Humman.  "It's just that, once you've reached the top tier of chess -- which I think is a more sensible thing to talk about than nonexistent Elo scores on a nonexistent line, the top tier is just the state of understanding all the core chess insights there are to know -- there's not much point in trying to compare yourself to others.  The complicated truth is merely that each top-tier player will be better in some places and worse in others, and any claim otherwise is just obviously false if you've ever played chess."

Tessa's face screwed up in thought.  "The reality that's more complicated than the big straight global line of Elo scores might look like... a function from every possible chessboard position, onto how likely your brain is to make each possible legal move from that position, with probabilities varying depending on how much sleep you've had.  Suppose we compare that whole function with Mr. Neumman's function, and compare how good are the probable moves you'd make versus him making.  On most chess positions, Mr. Neumann's move would probably be better.  We can imagine a comparison between those two vast functions, overlaid with vectors, little arrows, whose direction and length say how much better Mr. Neumann's move would probably be than yours, or rarely point the other way.  And then while the arrows don't all line up perfectly, they're not just random; ninety percent of them are pointing in the same direction, toward Mr. Neumann being better.  That's the detailed complicated actually-true underlying reality that explains why the Elo system works to make excellent predictions about who beats who at chess.  Down in actual reality there's lots of small skill-difference arrows, not perfectly aligned, but lined up in mostly the same direction as the imaginary big Elo-difference arrow, weighed up across the sort of chess positions that probably arise when you and Mr. Neumann play in practice."  Tessa sighed performatively.  "It really is a classic midwit trap, Mr. Humman, to be smart enough to spout out words about possible complications, until you've counterargued any truth you don't want to hear.  But not smart enough to know how to think through those complications, and see how the unpleasant truth is true anyways, after all the realistic details are taken into account."

"I should hardly think anyone ought to listen to you about that sort of matter," said Mr. Humman, "when you are hardly a top-tier chess-player yourself."  He smiled, then, with the satisfaction of having scored a truly searing point.

"What, the matter of whether or not it's epistemologically possible to sensibly say that one chess-player is stronger than another?" said Tessa.  "I don't think that being able to think that part through carefully is quite the same skill as knowing how to fork a king and queen, Mr. Humman."

"Why, of course it's the same," said Mr. Humman.  "You'd know that for yourself, if you were a top-tier chess-player.  The thing you're not realizing, young lady, is that no matter how many fancy words you use, they won't be as complicated as real reality, which is infinitely complicated.  And therefore, all these things you are saying, which are less than infinitely complicated, must be wrong."

"Look, Mr. Humman.  You may not be the best chess-player in the world, but you are above average.  People who show above-average ability at chess, usually but not always measure as having above-average ability at other cognitive tasks.  Your imaginary 'IQ score' that we infer from imperfect correlations like that, should be high enough that people with that 'IQ' can often comprehend ideas at this level of abstraction.  Or to say it in the shorthand people usually use in everyday life: 'You ought to be smart enough to understand this idea.'  If you'd just try to understand it, Mr. Humman!"

"Given that it's not actually true that chess ability runs off a single number floating over our heads," said Mr. Humman, "it is self-evidently dehumanizing to reduce a lifetime of chess-playing practice and effort and experience down into a single Elo score.  Like that's all a chess-player even is!  Like some players are just better than others!  It's obvious that the real reason why people resort to all this fancy math is just for the self-satisfaction of telling others:  I'm better!  You're worse!"

"Should I go around telling people that you admit you're no better than a 5-year-old at chess, given that you say no chess player is truly better than any other?" said Tessa.

"Oh, obviously I didn't mean it like that!" said Mr. Humman.  "I just mean that once you get to the level of top-tier chess-players, like me, there's no point in trying to compare us past there."

"Is there no level on which you can admit Mr. Assi was better than you at chess?" said the woman.  "Given that he was playing three people at once all day long, and I think beat every single one without one lost or drawn game."

"Well, the vast majority of the people he beat were not very good chess-players to begin with," said Mr. Humman, "unlike me.  But I did notice, and think that it was quite impressive, that Mr. Assi could play much faster chess than I could, if I needed to avoid blunders.  In a timed game with very little time, I would have made more of the sort of mistakes that a ten-year-old makes, and Mr. Assi would make fewer.  I also couldn't play three games simultaneously.  And so you see, young lady, by admitting that fact, I have fully proven my ability to 100% appreciate all of the advantages that Mr. Assi actually has, as a chess professional."

"Huh," she said.  "I guess I should give you some points for being able to imagine and admit to any way at all that Mr. Assi could be better than you.  Even if you made it be about the completely blatant, directly surface-visible fact of his speed, or the volume of chess-work he could output; rather than any slightly more abstract ideas, like how Mr. Assi's moves more effectively navigate the tree of possible chess positions."

"But of course," Mr. Humman continued, "all of that only matters under very artificial conditions imposed from outside, or as a contrived setup.  In real life, we both have time to think and avoid obvious blunders before we move, so there is not a very great difference in real life.  The reason I think it's fair to say that I'm genuinely better at chess than a 5-year-old is that the 5-year-old is probably having trouble remembering some of the rules, and hasn't learned all of the key ideas, like forks and skewers and pawn formations.  But once you learn all those key ideas and get some practice with them, what else could there be to learn?  In real life, two top-tier players have both learned every sort of key idea there is to know about chess, and can't learn them again.  What's left from there is fine practice and fine adjustments; though also, I agree, the further matter of speed."

"So you don't think there's also some sense in which Mr. Assi produces moves of... actually higher quality than yours," said Tessa.

"Why, I can't quite imagine how he could," said Mr. Humman.  "I didn't see him using any ideas or rules that I didn't know about.  For somebody to truly be better at chess than me, they'd need to produce some sort of miracle move that I didn't know was possible, and a miracle like that is contrary to the notion that chess has rules."

"You don't think there's any chess insights an International Master might possibly have picked up, that you don't know?"

"I can't think of any," said Mr. Humman.

"You know, Mr. Humman," said the woman, "I really think you'd be better off in life, if you figured out how to configure your emotions and personality in a way where you didn't need to occupy the ultimate top tier of chess-playing in order to grant yourself any respect at all.  Very few people can be chess champions of the world -- and even those champions, got there by playing a lot of chess games that they managed to enjoy before anyone acknowledged them as the world's top players.  I can see how it might rankle you to acknowledge that Mr. Neumann was reliably beating you at chess.  But would it invalidate your whole life to admit that a FIDE-recognized International Master can be just plain better?"

"There just isn't any such thing as 'better' in chess," said Mr. Humman.  "The right move in one game is just a wrong move in another, depending on who you're playing and what sort of luck you get from there.  I think I read once about a mathematician proving something like that mathematically; the no-free-lunch theorem, I think it was called, though it wasn't about chess."

"A ha ha, just a second, I need to text my friend back," the woman said, and hastily entered some keypresses into her cellphone.  A minute later she looked up again.  "Anyways!  Mr. Humman, I don't think theorems like the no-free-lunch theorem are supposed to apply to chess, or to the real world either.  They're more about proving that some non-chess-like setup doesn't have better or worse moves at all.  If those theorems applied to chess, you really would be exactly as good at chess as a five-year-old.  Or maybe a different way of putting it would be: there's no absolutely free lunch in a world of equal logical possibilities, but in a world of uneven realistic probabilities, a lunch can be pretty cheap.  If you tried applying those theorems to real-world situations, they'd say something like:  If every day for your whole life the charge of an electron has stayed constant, and so you bet your ten dollars against their ten million dollars that tomorrow the electron's charge will be the same, then here in the real world you'll win ten million dollars.  But you'll do worse in the logically possible world where winged monkeys swoop out of the sky and eat anyone who bets on that."

"Brilliant!" exclaimed Mr. Humman.  "That's exactly the sort of proof I mean.  Even if you think some chess move is the best chess move ever, what if in the real world you make that move and then a car runs you over?"

"Usually, in the real world, a car does not run me over," said the woman.

"But it could!" Mr. Humman said triumphantly.  "And that proves nobody can truly be better than anyone else at chess, and specifically, Mr. Neumann can't be generally better than me at chess, because a car could just run him over."

Tessa sighed.  "You know, even if somebody didn't understand the exact detailed math of something like a no-free-lunch theorem, you would really think that somebody could... just think about the thing someone is trying to proclaim that math implies, in an everyday sense... and see that informal claim doesn't match up with the sort of everyday life they could understand concretely?  Like, it would imply they weren't really any better at chess than a squirrel?  But I guess someone really does need to be far to the hooded-cloak side of the bellcurve from a midwit, before they get fast accurate math intuitions that fully reproduce the mental work of a based troglodyte."

"And the no-free-lunch theorem isn't the only piece of math I've heard about and you haven't," continued Mr. Humman.  "Like Ricardo's Law of Comparative Advantage, which says that you can always do better by having someone else help you, even if you think you're better at the job than they are, because it's easier when the job is split up among more people.  Always.  So you see, it's math itself that says that Mr. Assi could've played better chess if he'd accepted me as a partner.  If you think that sounds wrong, go study the math yourself --"

"Sorry, my friend just messaged me again on Discord and it sounds urgent," Tessa said, hastily keying some more words into her phone.  This time it was longer before she looked up again, though to be fair to her, Humman had been very wrong there.  "Anyway!  I've heard about Ricardo's Law, Mr. Humman," never mind when she'd heard about it.  "It's about how even if one country is more productive at everything than another country, they can still often benefit by trading --"

"Yes, like how Mr. Assi could've benefited by paying me to help him decide chess moves, even if he's a quicker chess-player than I am," said Mr. Humman.  "That's exactly what I said."

"It's not what that math says, Mr. Humman!  It's like -- one country can produce sausages with 1 hour of labor each, by hunting down buffalo and turning them into sausage, and can make sausage buns with 2 hours of labor, counting how long it takes to grow and mill grain.  And another country has actual machinery and can produce sausages with 2 minutes of labor, or buns with 1 minute of labor, even taking into account paying interest on the cost of machines.  Then even though the second country is more productive at everything, it can still benefit by shipping buns to the first country to trade for sausages one-to-one, which is a good trade for the first country too.  But the thing is, Ricardo's Law has all kinds of assumptions that it needs in the background, like the cost of shipping not being so high that it eats up all the gains from trade.  If one country is on Mars and the other is on Earth, the cost of rocket fuel would be way higher than the value of either sausages or sausage buns, if that was literally the stuff being traded.  Or if one country has some rotten sausages mixed in with their shipment, it might be too dangerous to buy from them, or too costly to check all their sausages by hand.  That's what it would be like for Mr. Assi to try to have you help him play chess!  Even if there was a chess possibility that he didn't have time to think about himself, the amount of time it would take him to explain to you what that chess question was, in enough detail to make that helpful, would be waaaay higher than the amount of time it would take him to answer that question himself.  His brain is doing the work of chess internally by talking to itself quickly and not just in words.  There aren't going to be questions that he can factor out and give to you in words and consider your answer in words, in less time than it takes Mr. Assi to think it through himself and arrive at a better answer than you'd give him.  Your brain's sausages and buns are both located on Mars, relative to his brain -- actually now that I try to talk about it, I don't see how this is the kind of setup that Ricardo's Law talks about at all, in the first place.  And that's even before considering how sometimes you'd give answers that Mr. Assi thought were terrible, unless he redid all your work himself."

"Well now you're just being insulting," sniffed Mr. Humman.  "I'm not a five-year-old who'll sometimes make mistakes about what the chess rules are, and I'm not a ten-year-old who moves pieces where they'll get captured right away.  What we're seeing here, young lady, is how your wrongness is like crystallization spreading through ice.  Your first mistakes just lead to more mistakes.  You think Mr. Assi is somehow a better chess-player than myself, instead of being good at different things and faster than me.  And now that's leading you to defy what Math Itself says about how I could help him play better chess, if he'd just work with me."

"Do you think Ricardo's Law says that any company can always do better by hiring any person on Earth as a new employee?" said Tessa.  "Because it sounds like that's how you're trying to overextend it."

"Of course not any new person," said Mr. Humman.  "But I would be a fine employee at any company that hired me!  Not one of their best contributors maybe -- not before I'd had a chance to learn my job as well as any other employee, to reach the top tier of skill for that job -- but of course I'd be a positive contributor.  It's not as if I'd make anything worse!  So yes, of course any company would do better by hiring me, than by not hiring me; I'm often surprised by how few companies seem to see that.  And it doesn't help to tell them it's a mathematical theorem, either."

The woman sighed.  "Let's change the subject."

"Fine by me," Mr. Humman said, and turned back to the soup shelf in the corner of the grocery, in which they'd been standing and arguing this whole time.  (That the grocery management did not object to this sort of behavior was part of how the grocery had become a civic institution of Skewers, WA on par with its chess club.)  "Have you seen any good... your generation doesn't really watch movies any more, does it.  Seen any good 30-second videos on Tocktick, or whatever it's called?  Or are they down to 20 seconds by now?"

"I don't actually watch TikTok videos either," said the woman.  "I, too, would like to die with relatively more of my brain intact.  Hm.  We probably shouldn't try to discuss politics, should we?"

"We really shouldn't," said Mr. Humman.  "It never ends well, either the discussions, or the politics themselves."

"And the state of the economy is probably also out."

"I wouldn't want to hurt a young person's feelings by raising that topic with them," said Mr. Humman.

"Yeah," she said.  "Well.  Have you read any good books lately?"

"There are no more good books," Humman said, picking up a can of meatball soup, examining the ingredients list for forbidden ingredients, and putting it down sharply again.  "The entire front wall of our Barnes and Noble is fiction about billionaire werewolves and the secret heirs of Faerie who get abducted by them."  Mr. Humman paused thoughtfully.  "I suppose it paints a grim picture when you put it all together.  Probably the world is coming to an end, don't you think?  And if it isn't, IT SHOULD BE."

"Well, by coincidence, that is sort of the topic of the book I'm reading now," said Tessa.  "It's about Artificial Intelligence -- artificial super-intelligence, rather.  The authors say that if anyone on Earth builds anything like that, everyone everywhere will die.  All at the same time, they obviously mean.  And that book is a few years old, now!  I'm a little worried about all the things the news is saying, about AI and AI companies, and I think everyone else should be a little worried too."

Mr. Humman snorted.  "My own extremely considered opinion, as someone older and wiser than you, is that this particular apocalypse prediction is wrong, and anyone ought to see at a glance that it's wrong -- sadly enough."  Mr. Humman laughed a little, at this humorous remark he'd just made.

"The authors don't mean it as a joke, and I don't think everyone dying is actually funny," said the woman, allowing just enough emotion into her voice to make it clear that the early death of her and her family and everyone she knew was not a socially acceptable thing to find funny.  "Why is it obviously wrong?"

"Because there's no such possible thing as 'super' intelligence," said Mr. Humman.  "It's got exactly the same sort of problem as saying that Mr. Assi is a better chess-player than I am -- as if he could beat me at any chess game, every time."

"I'm not sure a powerful alien intellect would need to beat every human at every mental contest every single time, in order to take over the world?" said Tessa.  "But also, I absolutely would bet on Mr. Assi to beat you, Mr. Humman, as close to every time as makes no difference.  Maybe other International Masters could see where he's got weaknesses, and try to exploit them.  That doesn't mean you can detect his weaknesses, or that your own strengths could beat his weaknesses.  That's pretty much what the authors warn would happen with humans and ASI."

"And like I keep trying to say, that's nonsense!" said Mr. Humman.  "Why believe that anything smarter than a human is possible?  It's just like how, once you know all the things there are to know about chess and become a top-tier chess-player like me, there isn't any way to be truly better at chess."

"Mr. Humman, you may not like to think about it, but you're not actually the level of chess player that Mr. Neumann is," said the woman.  "You might prefer not to stop and think about it, but it's true.  You going around saying that you and Mr. Neumann are both 'top-tier chess-players' doesn't make there be no difference between you -- to say nothing of the gap between you and Mr. Assi.  Well, similarly, you're not the same level of thinker as John von Neumann -- or Einstein, if you don't know who von Neumann was, although the geniuses alive at the time seemed to agree that von Neumann was scarier.  That should already be enough to warn you that you're not in the top tier of all possible thinking engines, and haven't pushed the bounds of possible cognitive power to their limits.  And then the gap between John von Neumann, and an ASI, could be much much wider."

"Intelligence is not a single line on a single spectrum," declared Mr. Humman.  "Reality is far more complicated."

"So there's no sense in which you're smarter than a squirrel?" she said.  "Because by default, any vaguely plausible sequence of words that sounds it can prove that machine superintelligence can't possibly be smarter than a human, will prove too much, and will also argue that a human can't be smarter than a squirrel."

"Oh, well, of course I'm smarter than a squirrel.  A squirrel doesn't have language, or the level of abstract thought needed to learn chess without it being an instinct.  But once your species has invented language and abstractions, it's reached the top tier of intelligence, which humans like myself and John von Neumann occupy together; and then there's no way to be truly any smarter than me and him."

"And you're not worried about the part where ASI could absorb the entire body of scientific literature in an hour and remember it perfectly, which, you know, even John von Neumann couldn't do.  You're not worried about how an ASI could have and create new senses for itself, new sensory modalities that help higher cognition with lower-level cognition, beyond what humans have in the way of vision and hearing and spatial visualization of 3D rotating shapes.  You're not worried about how it could split up into a thousand mutually telepathic instances of itself that shared memories and insights and learned skills and never forgot them.  You don't think that a mind like that, with detailed access to its own code and its own processes, could develop reflectivity that is substantially more powerful than the fragmentary and confused self-awareness that we humans use to think about thinking and organize our flailing thoughts.  You're not worried about an ASI's ability to fix the flaws it sees in itself and self-improve.  None of this strikes you as more of the same kind of jump that might distinguish a human brain from a chimpanzee brain?"

"All the improvements in a human brain over a chimpanzee brain just go into being able to use abstraction and language," explained Mr. Humman.  "And then once you can do that, you've got the last potent ability that any intelligence can ever acquire, having entered the top tier of sapience.  If we meet aliens from a billion light-years away, a billion years older than us and correspondingly more evolved, they will not really be any more intelligent than we are; we are already at the top.  Or I am, at least."

"I guess I have some trouble understanding on a visceral level how anyone could possibly, possibly believe that, though it is obvious that some people do," the woman said.  "You'd think that the part where the maximum human brain size is limited by the width of a woman's hips, and the adult brain has to run off twenty watts of power from eating fat and sugar, would be a hint about the further limits of possibility for brains the size of large buildings running off nuclear energy."

"I read a nice bit of science fiction by an author named Greg Egan, who called it the General Intelligence Theorem, based on an idea you've surely never heard about called Turing-Completeness; once you can simulate any possible process inside your own mind, that makes you as smart as it is possible to be, and you can't get any smarter.  If there were something smarter than you, you could just simulate it."  Mr. Humman smiled reminiscently.  "Now there was self-evidently a very smart man -- no smarter than me, of course, but much smarter than you -- which you can tell, because the things he says sound so validating and flattering."

Tessa didn't need to hurriedly consult Gemini in order to see the problem with that one.  "To say that Turing completeness defines the maximum level of intelligence would equally prove the human-equivalent intelligence of an unprogrammed CPU chip, a vacuum-tube computer from 1945, a sufficiently well-trained dog, Conway's Game of Life, some known small molecules, probably literally most small collections of small molecules if anyone put in some work into figuring out how to arrange them; and if you then point out the existence of memory bounds, why, human brains have those too.  An immortal human could, in principle, simulate an LLM with a trillion weights using a pen and paper; but that doesn't mean you'd come to understand everything the activations inside the LLM were reasoning about -- not any more than an immortal dog trained to implement a cellular automaton simulating out the neurons in your brain would have to learn chess first."

"Ah, well, I suppose I should've been more cautious about believing everything I read in science-fiction, then," said Mr. Humman, after several frantic mental tries failed to produce any possible way to defend that argument any further.  "And you, young lady, should consider the same caution."

"So you're not worried about the part where a machine superintelligence maybe thinks thousands of times faster, to the point where humans look to it like the barely moving statues from a 1000-to-1 slow-motion video."

"Oh," Mr. Humman said.  "Hm."  His mind could visualize that part, with a little effort.  "Well, in the end, that's all the better for humanity, isn't it?  If machine intelligences can do some scientific brain-work faster, it means we get more scientific breakthroughs, earlier.  Though of course, not with anything like a 1000-to-1 speedup.  There will still be a need for exactly as many experiments to be done as before, no fewer, and only human hands will be able to do those."

"I am maybe not as truly deeply acquainted with the depths of human history as some people are," said Tessa, "but when I read about the history of smithing, or the history of steam engines, it doesn't read to me like every good idea was invented, tested, and brought into production, as fast as it could possibly be thought up, over the course of human history.  There are technologies that rely on other technologies to develop.  It's hard to build a good steam engine without good steel.  But somebody has the next idea, like... one decade later, in history.  Not immediately.  You could take the AI algorithms that run on today's GPUs back to the year 2001 from before the age of deep learning, and they wouldn't do everything that today's GPUs can do with them, but they'd be able to economically useful things that actual 2001 AI couldn't do.  A superintelligence would invent those algorithms almost right away, if it was much smarter about computer science than humans.  Or think of how it is in biology, where it used to be the case that the only way to know how a new protein had folded up, was to make a bunch of that protein, and then do X-ray crystallography to it, and painstakingly interpret the results.  Nowadays you throw the DNA sequence into AlphaFold 3 and it immediately predicts how the protein will fold.  When people tried to get an AI model trained on bacteriophage DNA sequences to generate de novo bacteriophages, it got some that worked on the first try, it didn't need to do a ton of testing and refining to get to the point of having any successes.  And as for AI always needing human hands, I take it you haven't seen any of the recent videos of robots and androids?  Of the sort that just humans are building, after long hard struggles to invent the right software and hardware to test.  An ASI could build better robots than that, I'm pretty sure.  It could maybe build better biological humanoids to serve as hands, if it needed those; or downright Lovecraft-shoggoths to serve as hands, with cells that reproduce as fast as algae and then combine into larger bodies, in some much much more powerful version of how even tiny little AI models can figure out the structure of bacteriophages and...  Probably none of this is going to land on you, is it."

"I'm quite sure that if there were any possible body plan superior to the human form, or any way to make a biological creature more adept to serve as a superintelligence's hands, Nature in Her far greater wisdom would've invented all that already," said Mr. Humman.  "Even if some machine mind could invent its own robots, they would no doubt be better at doing some jobs than humans, and worse at doing others."

"Because we've... already got top-tier bodies for doing things... and no kind of body can be truly better than ours...?"

"Well-put!" exclaimed Mr. Humman.

"I don't understand what this kind of viewpoint has to say about... why it is that unarmed infantry troopers don't just charge straight at tanks, if the human body is already in the top-tier of military armanents," she said.

"It says that tanks are bad at driving themselves, and need human drivers," said Mr. Humman.

"I'm not sure how long that is going to stay true," said Tessa.  "In case you haven't heard about the whole thing with robotic cars."

"Well, then tanks are bad at building more tanks," said Mr. Humman.  "Unlike humans, which can make even more humans, that then go build tanks.  That is, in fact, why tanks have not already taken over the world economy, even if a naive person like you might've been impressed by their mighty armored treads.  Tanks are better at some things than humans, and worse at others; that is why they are unable to replace us.  That is how it will always be, with everything, forever.  If a billion-year-old civilization of aliens were to meet us, they wouldn't seem any different, except for maybe finally understanding that no top-tier species is truly superior to any other.  The aliens would be no smarter, they would have no better bodies, and there would be plenty of work in their economies for us to do -- to the point where there was no point in them trying to conquer the Earth or take our land away, when we could instead work that land ourselves, and trade with them.  The profit to the aliens would actually be greater that way, because they wouldn't have to birth and raise new workers."

"It would be a nice thought to imagine that the West could've gained just as much wealth from trading with existing Native American cultures, left intact, than by stealing their land and building a Western economy on it," said Tessa.  "I wish the world did work like that, and that there was never any financial reward at all for theft, murder, and genocide.  But I cannot say with a straight face that we live in that world.  If you imagine, say, modern Russia, coming across a portal to a parallel Earth with an early-hunter-gatherer-level Eurasian continent, it would just be true that Russia could make more money faster by shoving natives off the land and developing it themselves.  To choose to not murder a people or take their land, a sufficiently advantaged country has to care about something more than just wealth, to make that decision; it has to care about people."

"I suppose that may have been true back then," said Mr. Humman.  "But -- though it may be a bit impolitic to say it -- the original Native Americans did not possess a top-tier economy, like we moderns have now achieved.  It might be inconvenient for modern Russia to hire early hunter-gatherers right off and immediately to work in their economy; but that is because hunter-gatherers have not gone to modern schools, which produce the most adequate kind of employees that can exist, and take top-tier people like me over an absolute threshold of always being employable."

"The part that I am worried about," said the woman, "is an ASI that could, at the very least, almost trivially clone beings with the bodies of athletes and the brains of John von Neumann, and tweak their neurochemistry and brains to make them better slaves when appropriately raised from birth -- or you are a more powerful ASI that could figure out how to build entire new organisms -- and for that matter, new kinds of biology, that maybe initially get built by proteins but then aren't proteins at all -- like how proteins build bones that aren't made of protein, or how humans pour steel that isn't made of human flesh.  I worry that beyond that point, the superintelligent-designed optimal economy, full of factories that build parts that go into factories, and factories that build workers that build factories, does not optimally include any human alive today.  I worry that we would just slow down any part of a well-designed economy, where you tried to add a human; because we wouldn't tolerate the optimal heat or the optimal cold or the optimal radiation level, or because we'd need to eat every day instead of running off wall current, or because our hands wouldn't make fine enough motions quickly enough, or because we'd sometimes make mistakes, or because we'd think much too slowly, or above all because we wanted to get paid.  If ants could talk and trade with us and conformed reliably enough as employees, we'd probably find something in the world economy for ants to do!  But that's because human engineers are not good enough at biology to build better ants that don't need paying!"

"You have exactly described the outcome that I am utterly sure will never happen," said Mr. Humman.  "A human is a top-tier mind, armed with a top-tier body, made out of top-tier biology; and to pay us a comfortable wage produces a worker on the ultimate frontier of cost-effectiveness.  There could be nonhuman creatures that are better in some ways, and worse in others.  But nothing can be entirely better than a human -- not even a whole economy built out of specialized pieces, because then that whole economy is just a top-tier economy the same way that humans form a top-tier economy.  It will always make sense to employ individual humans, and trade with our collectives, and fit us into the system somewhere comfortable for us; because it is impossible to make any creature or any complete economical system of specialized components that is really better.  I nearly dirty my mouth by speaking such nonsensical words!"

"The same way that, if Mr. Assi wasn't so stubborn, he'd have realized how much it would benefit his chess-play to bring you along for pair games and hear out your advice, to shore up his own weaknesses," said the woman.

"Exactly!" exclaimed Mr. Humman.  "I'm sure that machine minds will be less stubborn and more humble than that awful fellow, when it comes to hearing out how very much I have to offer -- a top-tier existence like myself."

By a strange sort of coincidence -- if you don't take into account that conversations like that had played out all over the world, now and then and here and there, and so something like this was bound to happen to someone -- it was at that exact instant that a pair of tiny flying robots the size of mosquitos landed on the necks of Mr. Humman and Ms. Tessa, just above their respective carotid arteries, and they both fell over dead a few seconds later.

The End.

(Though that was not -- this author is humble enough to accept, and go on writing anyways -- an instance of the best possible, ultimate top-tier sort of literary ending.)