There seems to be a lack of emphasis in this market on outcomes where alignment is not solved, yet humanity turns out fine anyway. Based on an Outside View perspective (where we ignore any specific arguments about AI and just treat it like any other technology with a lot of hype), wouldn't one expect this to be the default outcome?
Take the following general heuristics:
This, if applied to AGI, leads to the following conclusions:
I would expect the average person outside of AI circles to default to this kind of assumption.
It seems like the only option that seems fully compatible with this perspective is
G. It's impossible/improbable for something sufficiently smarter and more capable than modern humanity to be created, that it can just do whatever without needing humans to cooperate; nor does it successfully cheat/trick us.
which is one of the lowest probabilities on the market. I'm guessing that this is probably due to the fact that people participating in such a market are heavily selected from those who already have strong opinions on AI risk?
Claims of the form "Yudkowsky was wrong about things like mind-design space, the architecture of neural networks (specifically how he thought making large generalizations about the structure of the human brain wouldn't work for designing neural architectures), and in general, probably his tendency to assume that certain abstractions just don't apply whenever intelligence or capability is scaled way up."
This is what I was gesturing at in my comments.
The claim about AI boxing I'm not sure about, but my understanding is that it's currently being debated ... (read more)
Arguably the most important topic about which a prediction market has yet been run: Conditional on an okay outcome with AGI, how did that happen?