You use the Copernican principle (along with the fact that there are almost certainly billions of planetary systems in our past light cone) to conclude that (1) it is unlikely that we're the only technological civilization in our past light cone. Then you go on to use the Fermi paradox. But why in your mind does the Fermi paradox not lead you to believe that we probably are the only civilization in our past light cone (in spite of the Copernican principle)?
In other words, aren't you cherry picking by letting your argument rely on the lack of any evidence of a civilization-destroying AI's having reached our solar system while acting as if your argument (and in particular the component of your argument I have labelled (1) above) is immune to the lack of any evidence of a (non-human) non-destroyed civilization's having reached our solar system? Do you claim that an AI that destroyed the civilization that created it (if such existed in our past light cone) would have been more likely to expand than a civilization that avoided being destroyed would have been?
And I don't yet see how the concept of a Great Filter throws any light on the question you are wrestling with. If there is a Great Filter that prevents almost every simple form of life from evolving into a galaxy-spanning civilization, then why not just conclude that humanity has already passed this Great Filter?
I suspect that the OP's argument implies that we should double down on priors which imply the Great Filter and are yet consistent with observations. Suppose that the Milky way has
On the other hand, if the probability that interstellar travel is impossible was
What I struggle to understand is how many bits of evidence already point a perfect Bayesian in the direction of interstellar travel being possible. In a manner similar to Yudkowsky, I suspect that the amount of evidence pointing towards known physics is at least on the order of thousands of bits.
Well, we do know that interstellar travel is possible, just currently very hard. It's a technological issue, not a physical one. I can certainly imagine slow travel, a few thousands km/s, being possible with nuclear rocket propulsion or other tech. Getting a spaceship-size object traveling at an appreciable fraction of the speed of light is a different story, anything like that would create observational artifacts visible from far away, so those probably aren't anywhere in the observed past light cone. But I am not sure how this is related to the evaluation of the odds of AIpocalypse.
All good questions! We might be the only recognizably technological civilization out there, any other "life" might occupy something we cannot even tell is life.
I guess my point is that, provided we are not the only ones like us to begin with, concluding that we are still the unique ones that passed the Great Filter makes us pretty unique, and the question becomes "why?". If AI x-risk is the great filter, then we would see the artifacts of it out there, unless the AGI magically stops growing somewhere between the current level and the observationally visible one.
A clever arguer wanting to persuade you that AI is dangerous can construct an argument much like yours: among the at least 10^22 stars in our past light cone, a decent number probably birthed a technological civilization. And maybe some of those civilizations had no interest in expanding. But surely some wanted to expand and to keep on expanding till they reached our solar system -- if they could have. The most like explanation for why an alien civilization in our past light cone failed in their desire to expand and to keep on expanding is that their whole civilization went extinct. There is something dangerous about the universe that destroys civilizations. We don't know for sure what that might be, but one strong possibility is that messing with AI is the primary destroyer of civilizations, so we should be more hesitant to mess with AI than we would have been if the skies were opaque and we had no idea other star system existed.
I'm not saying that that is a sound argument. It is not because it ignores the fact that after destroying the civilization that created it, an AI will tend to want to expand into space, too, just like a civilization would, and we see no signs of an AI's having reach our star system. But your argument is equally unsound because it ignores the fact that we see no signs of any AI-surviving civilization's having reached our star system.
The correct conclusion to draw from our observations of the cosmos (and from the lack of any signs of an alien visit to our planet) is that we were mistaken in assuming that other star systems are similar to our star system. It is fine to assume that those 10^22 stars (which are definitely real, no doubt about that) are like our star until we encounter enough evidence that they are in fact not like our star. And we have that evidence, e.g., all the evidence we have that allows us to get very concrete about the nature of interstellar travel and of a interstellar probe or space ship's arrival at a new suitable star system followed by exploitation of the new system's resources to build more probes.
I can imagine that an AI first destroys humanity, then poofs away before reaching for the stars, for sure. My point is that the window is quite narrow there.
I can also appreciate the argument that we are special and a space-faring civilization, artificial or otherwise, did not arise until now, or at least not in any visible to us ways, let alone a lightcone-consuming way. But that argument is not really new.
I might be missing your point...
Bluntly stated, my point is that the proper amount of weight to give to your argument for "why I am not too worried about AIpocalypse" is zero.
Extrapolating a straight line that far means visible cosmic consequences: a normal planet or a star rather suddenly starting to behave very much unlike what we expect from the known physics: growing very bright, or very dim, or disappearing completely.
Two objections to this.
1) Maybe Dyson spheres and the Kardashev scale are just good old sci-fi tropes and completely off the mark (same for computronium or hedonium). Maybe a superintelligence simply doesn't do that. We don't know. We might be squirrels imagining that a superintelligence ought to stockpile astronomical quantities of nuts, visible from kiloparsecs away.
2) Even granting Dyson spheres, the Kardashev scale, and the rest, our most complete catalogue of individually resolved stars in the Milky Way (Gaia DR3) covers on the order of 1% of them. Most of the rest are hidden behind interstellar dust. In the vast majority of cases, we simply couldn't tell the difference between a star occulted by a Dyson sphere and one obscured by dust. And even among the stars we do see, only a small fraction have been systematically searched for the thermal infrared excess a Dyson sphere should reemit at around 300K, the dedicated surveys (IRAS, then WISE with the G-HAT project) have screened tens of thousands of candidates, not hundreds of millions. As for stars outside our own galaxy, the fraction we can resolve individually is negligible.
Even stronger: if we saw a star go dark from a Dyson sphere, we'd probably be assimilated or swept aside almost immediately. Near-C probes are another likely consequence of a full singularity within our past light cone.
1) I agree, that is quite possible. Also, a Dyson sphere still radiates JUST AS MUCH OR NEARLY AS MUCH AS THE HOST STAR DOES, that's just physics. Best you can do is to shift the spectrum to the microwave region, masking it with CMB. But even there one would see odd luminosity bumps from specific directions. Subject to the known physics, which we have no reason to think is violated anywhere.
2) That is a good point. Though we do know where the dust is, from the observations, since it occults many stars at once in a very specific way. Though maybe you are right, I have not looked into it in enough detail. So, if your point is that Kardashev II is so rare that we do not see observational signatures of non-grabby aliens just living their lives, then yes, I guess it is not impossible. I have not checked the literature in the field recently. I guess there might be something like SAI out there that consumed its creators and is sitting inside one or many Dysonspheres, not a bubble of computronium expanding at near-light speed.
Thanks. I would add that my first point - that we should stay humble about what a superintelligence ought to do - also extends to how it ought to do it.
Suppose superintelligences do converge on the most extreme solution : a computronium bubble expanding at light speed. That still doesn't imply they would convert the entire content of their light cone into a uniform computational medium visible from parsecs away. Speaking as a non-physicist, my impression is that the structures we see in the universe are not arbitrary, they exist because they are stable equilibria. Matter at cosmological scales seems to end up in a fairly short list: planets and chemically bound bodies held together by electromagnetic forces, stars supported by nuclear fusion, neutron stars supported by degeneracy pressure, and at the end of the chain, black holes. A diffuse gas of computronium has no obvious mechanism to resist its own gravity in the long run, it would presumably either settle into one of these familiar structures or collapse into a black hole. To settle into one of these familiar object could imply that computronium's abundance is limited by physical constraints like temperature and pressure in the core of planets and stars. The black hole case is also interesting, because a black hole is theoretically the densest possible computer (Bekenstein), but everything it computes sits behind an event horizon, and recovering information from its Hawking radiation appears to require staggering amounts of computation in itself. So either way computronium could end up inhabiting - maybe in a weak proportion - the kinds of structures we already see, or hardly see (black holes).
Moreover, the Standard Model of cosmology tells us that most of the content of the universe is dark. Setting dark energy aside, there is a strong consensus that dark matter substantially outweighs ordinary matter. We don't know what dark matter actually is, and we cannot rule out that it would make as good a computational harware as ordinary matter. We think dark matter interacts with ordinary matter only gravitationally (or, at most, very weakly through some other channel) but we don't know whether dark matter interacts non-gravitationally with itself. If such self-interactions exist, they could plausibly be exploited for computation, much as we exploit electromagnetism in classical and quantum computers. If we accept agnosticism on this point, then, all else equal, the prior probability that computronium would be built out of dark matter is higher than that it would be built out of ordinary matter, simply because there is far more of it. We could already be inside a computronium bubble made of dark matter without realizing it.
If there is a Great Filter, I’d still rather humans be safe and in control when we get there. Just because the future of anything originating from Earth might be bounded doesn’t mean we still don’t want to capture that future, compared to letting AI convert the Earth to paperclips before the Filter hits.
By the way, there’s this cool map someone made with like ~100 solutions to the Fermi paradox, in case that helps you consider more possibilities: https://www.lesswrong.com/posts/ifW65CHb3d9NWFBvQ/fermi-paradox-solutions-map
1I don't disagree with any of it, my point is that the odds of the AI paperclipping the Earth and then randomly stopping without any observational consequences from far away are pretty low.
What if the Filter inevitably happens before you reach astronomical signatures, and doesn’t itself create a signature? Or the signature is brief enough, and civilizations uncommon enough, that 150 years is not enough time to catch an example.
That's definitely possible! And it is a worry. But there is no reason to worry specifically about AI x-risk, since there is no indication that an SAI will have this oddly specific behavior.
I have no good gears-level model of AI, and the expert views are all over the place (see AI Doc), so the only remaining argument is my physical intuition and a black-box view. Which, in this case, is based on two principles (best called rules of thumb):
Let's start with the latter: the main argument for x-risk, as I understand it, is that AI will become smarter than us in every way, becomes an agent with its own goals and will just take the Earth from us and use it for its own purposes. This seems like a reasonable possibility, given how things are going so far. Extrapolating a straight line that far means visible cosmic consequences: a normal planet or a star rather suddenly starting to behave very much unlike what we expect from the known physics: growing very bright, or very dim, or disappearing completely. Copernicanism says that if it can happen here, it probably happens in countless places elsewhere. And yet we do not see it. Why?
Hanson's Grabby Aliens is a proposed way out, where we do not see the aliens coming until they are upon us. This, too, does not survive even the modicum of Copernicanism: the Universe has been around for 14 billion years, our local group of galaxies for almost 10 billion years, and it contains over a trillion stars and is only 20 million light years across, and many stars there are as old as the Sun. So, if Grabby Aliens developed anywhere in the local group within the last 0.1% of the Universe's lifetime, we would have been eaten already. And maybe we have been, but then the argument is moot.
So we have a contradiction between the law of straight lines and Copernicanism: we expect visible effects of AIpocalypse elsewhere, while there are none. What gives? The Fermi paradox says that naive Copernicanism fails, and we are not like anyone else, to the degree where we cannot see anyone else. What gives? Either we are unique and the universe is not a good guide, or the straight lines stop being straight and peter out before Big Bad things happen.
There are still a fair few orders of magnitude between now and Astronomically Visible Big Bad. Probably between 3 (Kardashev I) and 13 (Kardashev II). For comparison, the energy consumption by humanity has increased by 7 orders of magnitude so far:
Credit: Gemini.
So, unless we are unique, some limiting factor will be at play somewhere between now and 5-7 orders of magnitude from now, a bit less than what the humanity has scaled so far.
What might be this limiting factor? Who knows. It may well be self-annihilation, but that is materially different from the predictions based on AI x-risk due to unbounded resource consumption growth by the indifferent amoral bots. So there is a big unknown ahead, and that means that the probability of x-risk, the proverbial p(doom) is not a useful number to wave around.
So there you have it: no good reason to worry about conventional x-risk, a lot of good reasons to worry about the Great Filter.