While AI may halt by many reasons, which I listed in my map of AI failure levels (http://lesswrong.com/lw/mgf/a_map_agi_failures_modes_and_levels/), it does not explain Fermi paradox. ((
Because before crash an AI may create non-evolving von Neumann probes. They will be safe against philosophical crisises, and will continue "to eat" universe. If many AIs crashed in the universe before us, one surely created self-replicating space probes. Why we don't see them?
That's a good point. Possible solutions:
AI just don't create them in the first place. Most utility functions don't need non-evolving von Neumann probes, and instead the AI itself leads the expansion.
AI crash before creating von Neumann probes. There are lots of destructive technologies an AI could get to before being able to build such probes. An unstable AI that isn't in the attractor zone of self-correcting fooms would probably become more and more unstable with each modification, meaning that the more powerful it becomes the more likely it is to destroy itself. von Neumann probes may simply be far beyond this point.
Any von Neumann probes that could successfully colonize the universe would have to have enough intelligence to risk falling into the same trap as their parent AI.
It would only take one exception, but the second and third possibilities are probably strong enough to handle it. A successful von Neumann probe would be really advanced, while an increasingly insane AI might get ahold of destructive nanotech and nukes and all kinds of things before then.
Any real solution of Fermi paradox must work in ALL instances. If we have 100 000 AIs in past light cone, it seems unplausible that all of them will fail the same trap before creating vNP.
Most of them will have stable form of intelligence like local "humans" which will able to navigate starships even after AI fails. So it will be like old school star navigation without AI. We will return to the world there strong AI is impossible and space is colonised by humanoid colonists. Nice plot, but where are way?
Another solution to FP is that most of new AIs fail to superAI predator which sends virus-like messages via some kind of space radio. The message is complex enough that only AI could find and read it.
Not necessarily all instances. Just enough instances to allow our observations to not be incredibly unlikely. I wouldn't be too surprised if out of a sample of 100 000 AIs none of them managed to produce successful vNP before crashing. In addition to the previous points the vNP would have to leave the solar system fast enough to avoid the AI's "crash radius" of destruction.
Regarding your second point, if it turns out that most organic races can't produce a stable AI, then I doubt an insane AI would be able to make a sane intelligence. Even if it had the knowledge to, its own unstable value system could cause the VNP to have a really unstable value system too.
It might be the case that the space of self-modifying unstable AI has attractor zones that cause unstable AI of different designs to converge on similar behaviors, none of which produce VNP before crashing.
Your last point is an interesting idea though.
In my second point I meant original people, who created AI. Not all of the will be killed during creation and during AI halt. Many will survive and will be rather strong posthumans from our point of view. Just one instance of them is enough to start intelligence wave.
Another option is that AI may create nanobots capable to self-replicate in space, but not to star travel. But they anyway will jump from one comet to another randomly and in 1 billion year (arox) will colonise all Galaxy. We could search for such relicts in space. They may be rather benign from the risk point, just like mechanical plants.
Another option is that the only way an AI could survive halt risks is: or becoming crazy or using very strange optimisation method of problem solving. In this case it may be here, but we could not recognise it because it behavior is absurd from any rational point of view. I came to this idea when I explored an idea if UFOs may be alien AI with broken goal system. (I estimate it to be less than 1 per cent true, because both premises are unlikely: that UFOs is something real and that Alien AI exist but crazy). I wrote about it in my controversial manuscript "Unknown unknowns as existential risks", p.90.
https://www.scribd.com/doc/18221425/Unknown-unknowns-as-existential-risk-was-UFO-as-Global-Risk
Another option is that the only way an AI could survive halt risks is: or becoming crazy or using very strange optimisation method of problem solving. In this case it may be here, but we could not recognise it because it behavior is absurd from any rational point of view. I came to this idea when I explored an idea if UFOs may be alien AI with broken goal system. (I estimate it to be less than 1 per cent true, because both premises are unlikely: that UFOs is something real and that Alien AI exist but crazy). I wrote about it in my controversial manuscript "Unknown unknowns as existential risks", p.90.
https://www.scribd.com/doc/18221425/Unknown-unknowns-as-existential-risk-was-UFO-as-Global-Risk
Many humans fear AI, indicating that other intelligent species might also fear it and might consider not developing AI. I think the great filter is more likely something that no one fears but most industrialized species do. For example, if (in a universe similar to ours) because of a weird quirk of physics that you don't discover until after you solved string theory, building a radio destroys your planet then radios would be the cause of the great filter.
Many civilizations may fear AI, but maybe there's a super-complicated but persuasive proof of friendliness that convinces most AI researchers, but has a well-hidden flaw. That's probably a similar thing to what you're saying about unpredictable physics though, and the universe might look the same to us in either case.
Now your comment is visible. I could suggest that complexity of x-risks problem could play the same role. We know around 100 more or less different x-risks scenarios now - and many more are probably unknown. I mean that (may be, hope not) there are so many different x-risks that any civilization will perish.
If x-risks have destroyed <100 civilizations in our galaxy you are probably right. If they have destroyed more than, say, 10,000 my suggestion probably is.
While I agree with your logic, my point should be said as following:
100 risks all happen together in short time period and interact non-lineary with each other. This results in incalculable complexity of the task to prevent them and no civilization is able to solve this complexity in such a short time. If risks were separated in time and no interacting your point is more valid.
This complexity results in trade-offs like: we need safe AI as soon as possible to prevent nano and bio risks, but to create really safe AI we need as much time as possible. There are many such trade-offs and also there are higher levels of complexity of problems than trade-offs.
For civilizations with no singleton I agree with your logic. But there might be/have been civilizations where one king is the absolute leader and it is able to block research into whatever technologies it wants. What kind of great filter destroys an industrialized planet run by King Lee Kuan Yew who has a 1000 (earth) year lifespan?
I am not sure that exactly this is FP solution. Personally I am more incline to Rare Earth solutions.
But I could suggest that may be the complexity of risks problem is so complex that no simple measures like banning most technologies will work. The weak point is that one von Neumann Probe is enough to colonise all visible universe. So there should be something which prevents civilizations from vNPs creation.
There's no paradox, there's just people plugging in guesses for numbers which almost certainly wildly exaggerate the likelihood of intelligent life arising.
It took evolution 75% of its available runtime to produce intelligent life (which itself was likely the product of a particular runaway process which has happened exactly once on this planet), with a number of near-misses that would probably stagger the imagination if we knew about them all (indeed, the few near-misses we do know about -do- stagger the imagination, and lead people to believe religion might have something to it after all).
almost certainly wildly exaggerate the likelihood of intelligent life arising.
Or that intelligence existing for a long time necessarily implies interstellar spread.
...any advancing civilization would likely begin developing AI. ... AI development is a natural step in the path of progress
This is a very strong claim to be making from a sample size of one.
For hypothesis to hold AI needs to:
Seems a lot of presumptions, with no good arguments for any of them?
On the other hand I don't see, why AI that does spread can not be a great filter. Lets assume:
If it is true, we should find ourselves surprisingly early in the history of Universe. But if we consider that frequency of gamma-ray bursts is quickly diminishing, and so we could not be very early, because there were so many planet killing gamma-bursts, these two tendencies may cancel each other and we are just in time.
Also, what if intelligent life is just a rare event? Like not rare enough to explain Fermi paradox by itself, but rare enough, that we could be considered among earliest and therefore surprisingly early in the history of universe? Given how long universe will last, we actually are quite early: https://en.wikipedia.org/wiki/Timeline_of_the_far_future
(Retracted: If we take total number of stars that will be ever created we are somewhere in first 7 per cent (if I remember correctly - can't find a link), so we are early, but no surprisingly early)
Update: I was wrong. we are surprisingly late. 95 per cent stars is already created. http://www.wired.com/2012/11/universe-making-stars/
Update 2: but when the Sun was born it was exactly 50 per cent of stars were already born. It is strong argument for rare earth.
The Fermi paradox has been discussed here a lot, and many times it has been argued that AI cannot be the great filter, because we would observe the paperclipping of the AI just as much as we would observe alien civilizations. I don't think we should totally rule this out though.
It may very well be the case that most unfriendly AI are unstable in various ways. For instance imagine an AI that has a utility function that changes whenever it looks at it. Or an AI that can't resolve the ontological crisis and so fails when it learns more about the world. Or maybe an AI that has a utility function that contradicts itself. There seem to be lots of ways that an AI can have bugs other than simply having goals that aren't aligned with our values.
Of course most of these AI would simply crash, or flop around and not do anything. A small subset of them might foom and stabilize as it does so. Most AI developers would try to move their AI from the former to the latter, and in doing so may pass through a space of AI that can foom to a significant degree without totally stabilizing. Such an AI might become very powerful, but exhibit "insane" behaviors that cause it to destroy itself and its parent civilization.
It might seem unlikely that an "insane" AI could manage to foom, but remember that we ourselves are examples of systems that can use general reasoning to gain power while still having serious flaws.
This would prevent us from observing either alien civilizations or paperclipping, and is appealing as a solution to the Fermi paradox because any advancing civilization would likely begin developing AI. Other threats that could arise after the emergence of civilization probably require the civilization to exhibit behaviors that not all civilizations would. Just because we threatened each other with nuclear annihilation doesn't mean all civilizations would, and it only takes one exception. But AI development is a natural step in the path of progress and very tricky. No matter how a civilization behaves, it could still get AI wrong.
If this does work as a solution, it would imply that friendliness is super hard. Most of the destroyed civilizations probably thought they had it figured out when they first flipped the switch.