Statement of Support for "If Anyone Builds It, Everyone Dies"
18Buck
2Liron
4niplav
4Zack_M_Davis
3Ben Pace
0Liron
4Ben Pace
2Garrett Baker
New Comment
- I think the book's thesis is basically right — that if anyone builds superintelligent AI in the next decade or two, it'll have a terrifyingly high (15%+) chance of causing everyone to die in short order
I think this is an absurd statement of the book's thesis: the book is plainly saying something much stronger than that. How did you pick 15% rather than for example 80% or 95%?
I am basically on the fence about the statement you have made. For reasons described here, I think P(human extinction|AI takeover) is like 30%, and I separately think P(AI takeover) is like 45%.
- I think the world where the book becomes an extremely popular bestseller is much better on expectation than the world where it doesn't
I think this is probably right, but it's unclear.
- I generally respect MIRI's work and consider it underreported and underrated
It depends on what you mean by "work". I think their work making AI risk arguments at the level of detail represented by the book is massively underreported and underrated (and Eliezer's work on making a rationalist/AI safety community is very underrated). I think that their technical work is overrated by LessWrong readers.
Yeah there’s always going to be a gray area when everyone is being asked if their complex belief-state maps to one side of a binary question like endorsing a statement of support.
I’ve updated the first bullet to just use the book’s phrasing. It now says:
- I think the book's thesis is likely, or at least all-too-plausibly right: That building an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, will cause human extinction.
"If Anyone Builds It, Everyone Dies" is more true than not[1].
Personally my p(doom)≈60%, which may be reduced by ~10%-15% by applying the best known safety techniques, but then we've exhausted my epistemic supply of "easy alignment worlds".
After that is the desert of "alignment is actually really hard" worlds. We may get another 5% because mildly smart AI systems refuse to construct their successors because they know they can't solve alignment under those conditions.
So the title of that book is more correct than not. I think the AI-assisted alignment feels the most promising to me, but also reckless as hell. Best is human intelligence enhancement through some kind of genetech or neurotech. Feels like very few people with influence are planning for "alignment is really hard" worlds.
I nevertheless often dunk on MIRI because I would like them to spill more on their agent foundations thoughts, and because I think the arguments don't rise above the level of "pretty good heuristics". Definitely not to the level of "physical law" which we've usually used to "predict endpoints but not trajectories".
The statement, not the book. The book was (on a skim) underwhelming, but I expected that, and it probably doesn't matter very much since I'm Not The Target Audience. ↩︎
I could be described as supporting the book in the sense that I preordered it, and I bought two extra copies to give as gifts. I'm planning to give one of them tomorrow to an LLM-obsessed mathematics professor at San Francisco State University. But the reason I'm giving him the book is because I want him to read it and think carefully about the arguments on the merits, because I think that mitigating the risk of extinction from AI should be a global priority. It's about the issue, not supporting MIRI or any particular book.
1(I agree-reacted to the statement, because I think polling is often a very reasonable default activity. I am not upvoting the post, especially not for a goal of turning the frontpage post list into a space to signal the positions of LW, rather than for interesting and novel arguments and/or well written explanations.)
(Added: To be clear Liron did not ask for upvotes; but the post talked about what gets upvoted on frontpage, and then posted the poll claim, which seemed suggestive to me of a standard terrible failure mode for discussion spaces; so I want to disclaim upvoting in order to make the frontpage signal positions to low-engagement LW readers.)
I don't think LessWrong is the right venue for this, or really, I don't want it to become that venue. I basically agree with Steven Byrnes here, but generalized to also include rallying others to agree to stuff. I think its reasonable to make a post arguing people should publicly declare their stances on such things.
I think this sort of thing skirts too close to group-think.
Further, I don't know what it means to "support" a book. Does the book need help? How can that be? It's a book! See also: Ethnic Tension And Meaningless Arguments, in particular
A question: are you pro-Israel or pro-Palestine? Take a second, actually think about it.
Some people probably answered pro-Israel. Other people probably answered pro-Palestine. Other people probably said they were neutral because it’s a complicated issue with good points on both sides.
Probably very few people answered: Huh? What?
This question doesn’t fall into any of the three Philosophy 101 forms of argument. It’s not a question of fact. It’s not a question of particular moral truths. It’s not even a question of policy. There are closely related policies, like whether Palestine should be granted independence. But if I support a very specific two-state solution where the border is drawn upon the somethingth parallel, does that make me pro-Israel or pro-Palestine? At exactly which parallel of border does the solution under consideration switch from pro-Israeli to pro-Palestinian? Do you think the crowd of people shouting and waving signs saying “SOLIDARITY WITH PALESTINE” have an answer to that question? [...] The two-step works like this. First, load “religion” up with good karma by pitching it as persuasively as possible. “Religion is just the belief that there’s beauty and order in the universe.”
Wait, I think there’s beauty and order in the universe!
“Then you’re religious too. We’re all religious, in the end, because religion is about the common values of humanity and meaning and compassion sacrifice beauty of a sunrise Gandhi Buddha Sufis St. Francis awe complexity humility wonder Tibet the Golden Rule love.”
Then, once somebody has a strongly positive General Factor Of Religion, it doesn’t really matter whether someone believes in a creator God or not. If they have any predisposition whatsoever to do so, they’ll find a reason to let themselves. If they can’t manage it, they’ll say it’s true “metaphorically” and continue to act upon every corollary of it being true.
I know you clarify what you mean by "support" later, and of course you are committing a much smaller sin than the religious person in the example, but if someone said the "statement of support" you outline, what they are literally saying is meaningless. That statement can only be understood in the context of a political game they're playing with the rest of society.
Now that political game may be necessary and useful, but I'd rather LessWrong not be a battlefield for such games, and remain the place everyone can go to to think, not fight.