Let me try addressing your comment more bluntly to see if that helps.
Your complaint about Klurl's examples are that they are "coincidentally" drawn from the special class of examples that we already know are actually real, which makes them not fictional.
No, Klurl is not real. There are no robot aliens seeding our planet. The fictional evidence I was talking about was not that Earth right now exists in reality right now, it was that Earth right now exists in this story specifically at the point it was used.
If you write a story where a person prays and then wins the lottery as part of a demonstration of the efficacy of prayer, that is fictional evidence even though prayer and winning lotteries are both real things.
If you think that the way the story played out was misleading, that seems like a disagreement about reality, not a disagreement about how stories should be used.
No, I really am claiming that this was a misuse of the story format. I am not opposed to it because it's not reality. I am opposed to it because the format portends that the outcomes are illustrations of the arguments, but in this case the outcomes were deceptive illustrations.
If Trapaucius had arrived at the planet to find Star Trek technology and been immediately beamed into a holding cell, would that somehow have been less of a cheat, because it wasn't real?
It would be less of a cheat in the sense that it would give less of a false impression that the arguments were highly localizing, and in that it would be more obvious that the outcome was fanciful and not to be taken as a serious projection. But it would not be less of a cheat simply in the sense that it wasn't real, because my claim was never that this was cheating for using a real outcome.
I stand by what I said, but I don't want to argue about semantics. I would not have allowed myself to write a story this way.
The Star Trek claim is a false dichotomy. One could choose to directly show that the underspecified parts are underspecified, one could choose to show many examples of the ways this would near-miss, one could simply not write oneself into this corner in the first place. And in the rather hard to believe counterfatual that Yudkowsky didn't feel capable to make his story without such a contrivance, he could have just used a different frame, or a different format, or signposted the issue, or done some other thing instead.
"One does not live through a turn of the galaxy by taking occasional small risks."
I'll admit to this that the author being Yudkowsky heavily colored how I read this line. He has repeatedly, strongly taken the stance that AI risk is not about small probabilities, he would not be thinking so much about AI risk if his probability were order-1%, people who do care about order-1% risks are being silly, etc. There are lots of quotes but I'll take the first one I found on a search, not because it's the closest match but that it's the first one I found.
But the king of the worst award has to go to the Unironical Pascal's Wager argument, imo - "Sure the chances are tiny, but if there's even a tiny chance of destroying the lightcone..."
— https://x.com/ESYudkowsky/status/1617903894960693249
I do not know if I'm being unfair or generous to Yudkowsky to dismiss this defense for this reason. Regardless, I will.
I will say that the very next sentence Klurl states is,
"And to call this risk knowably small, would be to claim to know far too much."
and indeed I think this is an example where the literary contrivance hides the mistake. If the author wasn't forcing his hand, the risk would have been small. The coincidence they were in was unlikely on priors and not narrowed into from the arguments given.
—
What examples are you thinking of here?
It's obvious that human learning is exceptional, but I don't think Klurl's arguments even served to distinguish the rock sharpening skill from beaver dams, spider webs or bird nests, never mind the general set of so-termed ‘tool use’ in the wild. Stone tools aren't specific to humans, either, though I believe manufactured stone tools are localized to hominids, for example Homo floresiensis as a meaningfully distinct and AFAIK not ancestral cousin species.
Related but distinct, I'll draw specific attention to ants, which have a fascinating variety of evolutionary behaviours, including quite fascinating trap making with a cultivated fungus. Obviously not a generalizably intelligent behaviour, but yet Klurl did not even ask that of humans. (On an even less related note, Messor ibericus lays clones of Messor structor as part of their reproductive cycle, which is fascinating and came to mind a lot when reading the sections about stuff evolution supposedly can't solve because it, per the accusation, operates through one specific reproductive pathway.)
If this was presented as a piece of fiction first, sure, ‘bad for verisimilitude’. But Yudkowsky prefaces, best considered as nonfiction with a fictional-dialogue frame. When I consider it in that light, it's more than a problem of story beats, it's cheating evidence into play, it's an argumentative sleight of hand, it's generalizing from fictional evidence. I think it's misleading as to how strongly refutations actually hold, both in the abstract, and also as directly applied to the arguments Yudkowsky is defending in practice.
To the first point, I hope it was clear I'm not defending Trapaucius here. The story is maybe unfair to the validity of some of Trapaucius' arguments, but not that unfair, they were net pretty bad.
We have lots of examples of radiators in space (because it's approximately the only thing that works), and AFAIK micrometeor impacts haven't been a dealbreaker when you slightly overprovision capacity and have structural redundancy. I don't expect you'd want to spend too much on shielding, personally.
Not trying to claim Starcloud has a fully coherent plan, ofc.
It's not that complex in principle: you use really big radiators.
If you look at https://www.starcloud.com/'s front page video, you see exactly that. What might look like just a big solar array is actually also a big radiator.
AFAICT it's one of those things that works in principle but not in practice. In theory you can make really cheap space solar and radiator arrays, and with full reuse launch can approach the cost of propellant. In practice, we're not even close to that, and any short term bet on it is just going to fail.
Klurl is making a combination of narrow predictions, that for the most part aren't didactically justified.
By the framing, with the setup transparently referring to a real world scenario, it's easy for us to import a bunch of assumptions and knowledge about how the world described actually turns out, but Klurl doesn't have arguments that hang on this knowledge, and the story doesn't show why their prediction is precisely this, precisely now. This is especially blunt when Klurl gets to be very clever for pointing at Trapaucius' overly specific arguments, which we know as omniscient observers are falsified, but when Klurl makes overly specific arguments that happen to be right in this scenario but don't apply in generality to life on earth, the story doesn't in turn illustrate how this is in a natural class of errors.
I'll list some particular questions that felt unanswered. Again, because this points at reality it's very easy for me to gesture at why the world is as it is, but to the story's characters the entire branch of natural evolution is a novelty. As an illustration of how to argue about a position, it disproves too much: find any heuristic argument that disagrees with your point, find a known counterexample, have a character present the argument about that counterexample, have another character scoff at them for making it, reveal that it was reality all along, moral of the story is that scoffing at that argument is correct.
...Anyway, a list:
- Why was Klurl paying specific attention to the rock-sharpening skill, when evolution has a great many more much more impressive natural feats?
- Why was this specifically noticed in humans, rather than other animals that do this?
- Why did Klurl think this was time-urgent, when the vast majority of the time this would be a bad heuristic?
- Why was Klurl assuming humans could solve nuclear engineering but not that evolution could?
- Where comes the assumption that fleshlings were the kind of creature that would sustain attempts to circumvent its coordination failures, when there was no evidence of it presented?*
- Where comes the assumption that evolution wasn't the kind of creature that would sustain attempts to circumvent its coordination failures, when there was much evidence of it presented?
- What is the fairness of the selected human specifically being a counterexample to the proposed strategy, when even among their class few would have been?
I'm going to stop listing here because I don't want to reskim over the whole thing, and I don't think the point is missing much for lack of completeness.
*There was evidence given for some thing at some point solving hard problems. The claim that these were humanlikes was random.
How much could LeelaPieceOdds be improved, in principle? I suspect a moderate but not huge amount. An easy change would be to increase the compute spent on the search.
A few points of note, stated with significant factual uncertainty: Leela*Odds has already improved to the tune of hundreds of elo from various changes over the last year or so. Leela is trained against Maia networks (human imitation), but doesn't use Maia in its rollouts, either during inference or training. Until recently, the network also lacked contempt, so would actually perform worse past a small number of rollouts (~1000), as it would refute its best ideas. AFAIK Maia is stateless and doesn't know the time control. Last I checked, only Knight and Queen odds games were trained, and IIUC while training only played as white, so most of these games require generalising its knowledge from that.
2I think this is a misuse of the format. Klurl here is fundamentally presented as correct by coincidence. Like, uh,
In which the issue is contextualized
Oneicus and Twoicus were discussing the numberonica.
“I'm worried about the numberonica,” says Twoicus. “I'm concerned that next value produced by the random quantum circuit will be 8,888,888,888, which is a number we both dislike.”
“It won't though,” says Oneicus.
“Your reasons for believing it won't are silly,” Twoicus observes.
“It is improbable,” argues Oneicus. “As fact of the matter, I generated a great many random quantum circuits, and not one of them returned 8,888,888,888.”
“You are assuming that your sampling of random circuits is representative of the random circuit that this one is drawn from,” rejects Twoicus.
Over the course of many thousands of words, Oneicus presents increasingly unhinged arguments for why the number isn't 8,888,888,888, and indeed even should instead be precisely 147.
“[dismissive noises]” says Twoicus, because the two are bound inexorably to the demands of the allegorical format.
In which the issue is demonstrated
The numberonica dings, and displays its value: 8,888,888,888.
“...So it is, Twoicus. How did you come to know this would be so, and why did you not state this knowledge during the conversation, which was plenty many words long to hold it?” Onicus asks.
“I have already presented the arguments relevant to the conversation,” Twoicus indulges, “namely, a refutation of your arguments, which you must now agree are silly.”
“Why you little—” Oneicus expresses as a surprisingly muffling paper bag is placed over their head.
“This is not how arguments work.” Threeicus dictates in Oneicus' place. “Reality does not care how much more respectable you have demonstrated yourself to be than your opposition. Narrow beliefs form from positive knowledge.”
“So you say,” Twoicus bristles, “and yet.”
“Indeed,” Threeicus admits. “Therefore not.”
“Therefore wha—
To the first part: yes, of course, my claim isn't that anything here is axiomatically unfair. It absolutely depends on the credences you give for different things, and the context you interpret them in. But I don't think the story in practice is justified.
This is indeed approximately the source of my concern.
I think in a story like this if you show someone rapidly making narrow predictions and then repeatedly highlight how much more reasonable they are than their opponent as a transparent allegory for your narrow predictions being more reasonable than a particular bad opposing position from a post signposted as nonfiction inside a fictional frame, there really is no reasonable room to claim that actually people weren't meant to read things into the outcomes being predicted. Klurl wasn't merely making hypothetical examples, he was acting on specific predictions. It is actually germaine to the story and bad to sleight-of-hand away that Klurl was often doing no intellectual work. It is actually germaine to the story whether some of Trapaucius' arguments have nonzero Baeysean weight.
The claim that no simple change would have solved this issue seems like a failure of imagination, and anyway the story wasn't handed down to its author in stone. One could just write a less wrong story instead.