Relitigating the Race to Build Friendly AI
25habryka
8TAG
6jessicata
2habryka
6jessicata
2habryka
2jessicata
4habryka
2jessicata
11habryka
20Steven Byrnes
6Wei Dai
6Max H
6Wei Dai
New Comment
MIRI's plan, to build a Friendly AI to take over the world in service of reducing x-risks, was a good one.
How much was this MIRI's primary plan? Maybe it was 12 years ago before I interfaced with MIRI? But like, I have hung out with MIRI researchers for an average of multiple hours a week for something like a decade, and during that time period the plan seemed to basically always centrally be:
- Try to make technical progress on solving the alignment problem
- While trying to create a large public intellectual field that can contribute to solving that problem
- While trying to improve the sanity of key decision-makers who will make a bunch of high-stakes decisions involved in AGI
This also seems to me like centrally the strategy I picked up from the sequences, so it must be pretty old.
There was a period of about 4-5 years where research at MIRI pivoted to a confidential-by-default model, and it's plausible to me that during that period, which I understand much less well, much more of MIRI's strategy was oriented around doing this.
That said, it seems like Carl Shulman's prediction from 14 years was born out pretty well:
If we condition on having all other variables optimized, I'd expect a team to adopt very high standards of proof, and recognize limits to its own capabilities, biases, etc. One of the primary purposes of organizing a small FAI team is to create a team that can actually stop and abandon a line of research/design (Eliezer calls this "halt, melt, and catch fire") that cannot be shown to be safe (given limited human ability, incentives and bias).
After MIRI did a bunch of confidential research, possibly in an attempt to maybe just build an aligned AI system, they realized this wasn't going to work, then did a "halt, melt, and catch fire" move, and switched gears.
Rereading some of the old discussions in the posts you linked, I think I am more sold than I was previously that this was a real strategic debate at the time, and a bunch of people tried to argue in favor of just going and building it, and explicitly against pursuing strategies like human intelligence augmentation, which now look like much better bets to me.
To their credit, many of the people did work on both, and were pretty clear that they really weren't sure whether the "solving the problem head on" part would work out, and that they thought it would be reasonable for people to pursue other strategies, and that they themselves would pivot if that became clear to them later on. Eliezer, in a section of a paper you quoted yourself 14 years ago he says:
I do not assign strong confidence to the assertion that Friendly AI is easier than human augmentation, or that it is safer. There are many conceivable pathways for augmenting a human. Perhaps there is a technique which is easier and safer than AI, which is also powerful enough to make a difference to existential risk. If so, I may switch jobs. But I did wish to point out some considerations which argue against the unquestioned assumption that human intelligence enhancement is easier, safer, and powerful enough to make a difference.
Like, IDK, this really doesn't seem like particularly high confidence, and while I agree with you that in-retrospect you deserve some Bayes-points for calling this at the time, I don't think Eliezer loses that many, as it seems like all-throughout he proclaimed substantial probability on your perspective being more right here.
MIRI’s plan, to build a Friendly AI to take over the world in service of reducing x-risks, was a good one.
How much was this MIRI’s primary plan?
It was Yudkowsky's plan before MIRI was MIRI
http://sl4.org/archive/0107/1820.html
"Creating Friendly AI"
https://intelligence.org/files/CFAI.pdf
Both from 2001.
1What about the "Task AGI" and "pivotal act" stuff? That was at the very least, advising others to think seriously about using aligned AI to take over the world, on the basis that the world was otherwise doomed without a pivotal act. Then there was the matter of how much leverage MIRI thought they had as an organization, which is complicated by the confidentiality.
What about the "Task AGI" and "pivotal act" stuff?
Plausible! Do you have a link handy? Seems better for the conversation to be grounded in an example, and I am not sure exactly which things you are referencing here.
On Arbital. Task directed AGI and Pivotal act.
Offline, at MIRI there were discussions of possible pivotal acts, such as melting all GPUs. I suggested "what about using AI to make billions of dollars" and the response was "no it has to be much bigger than that to fix the game board". There was some gaming of e.g. AI for uploading or nanotech. (Again, unclear how much leverage MIRI thought they had as an organization)
Hmm, maybe I am misunderstanding this.
The "Task AGI" article is about an approach to build AGI that is safer than building a sovereign, published, on the open internet. I do not disagree that MIRI was working on trying to solve the alignment problem (as I say above, that is what two of the bullet points of my summary of their strategy are about), which this seems to be an attempt at making progress on. It doesn't seem to me to be much evidence for "MIRI was planning to build FAI in their basement". Yes, my understanding is that MIRI is expecting that at some point someone will build very powerful AI systems. It would be good for them to know how to do that in a way that has good consequences instead of bad. This article tries to help with that.
The "Pivotal Act" article seems similar? I mean, MIRI is still working on a pivotal act in the form of an international AI ban (subsequently followed maybe with an intelligence augmentation program). I am working on pivotal acts all day! It seems like a useful handle to have. I use it all the time. It does seem to frequently be misunderstood by people to mean "take over the world", but like, there is no example in the linked article of something like that. The most that the article talks about is:
- a limited Task AGI that can:
- upload humans and run them at speeds more comparable to those of an AI
- prevent the origin of all hostile superintelligences (in the nice case, only temporarily and via strategies that cause only acceptable amounts of collateral damage)
- design or deploy nanotechnology such that there exists a direct route to the operators being able to do one of the other items on this list (human intelligence enhancement, prevent emergence of hostile SIs, etc.)
Which really doesn't sound much like a "take-over-the-world" strategy. I mean, the above still seems to me like a good plan that in as much as a leading lab has no choice but to pursue AGI as a result of an intense race, I would like them to give it a try. Like, it seems terribly reckless and we are not remotely on track to doing this with any confidence, but like, I am in favor of people openly publishing things that other people should do if they find themselves building ASI. And again the above bullet lists also really don't sound like "taking over the world", so I still have trouble connecting this to the paragraph in the OP I take issue with.
I suggested "what about using AI to make billions of dollars" and the response was "no it has to be much bigger than that to fix the game board". There was some gaming of e.g. AI for uploading or nanotech. (Again, unclear how much leverage MIRI thought they had as an organization)
None of these sound much like "taking over the world"? Like, yes, if you were to write a paper or blogpost with a plan that allowed someone to make a billion dollars with AI, that seems like it would basically do nothing, and if anything make things worse. It does seem like helpful contributions need to be of both a different type signature, and need to be much bigger than that.
It doesn’t seem to me to be much evidence for “MIRI was planning to build FAI in their basement”
I didn't say that
The “Pivotal Act” article seems similar? I mean, MIRI is still working on a pivotal act in the form of an international AI ban (subsequently followed maybe with an intelligence augmentation program). I am working on pivotal acts all day!
At the time it was clear MIRI thought AGI was necessary for pivotal acts, e.g. to melt all GPUs, or to run an upload. I remember discussing "weak nanotech" and so on and they didn't buy it, they thought they needed aligned task AGI to do a pivotal act.
Which really doesn’t sound much like a “take-over-the-world” strategy.
Quoting task AGI article:
The obvious disadvantage of a Task AGI is moral hazard - it may tempt the users in ways that a Sovereign would not. A Sovereign has moral hazard chiefly during the development phase, when the programmers and users are perhaps not yet in a position of special relative power. A Task AGI has ongoing moral hazard as it is used.
So this is acknowledging massive power concentration.
Furthermore, in context of the disagreement with Paul Christiano, it was clear that MIRI people thought there would be a much bigger capability overhang / FOOM, such that the system did not have to be "competitive", it could be a "limited AGI" that was WAY less efficient than it could be, because of a pre-existing capability overhang versus the competition. Which, naturally, goes along with massive power concentration.
I didn't say that
Wait, you didn't? I agree you didn't say "basement" but the section of the OP I am responding to is saying:
MIRI's plan, to build a Friendly AI to take over the world
And then you said:
What about the "Task AGI" and "pivotal act" stuff? [Which is an example of MIRI's plan to build a Friendly AI to take over the world]
The part in square brackets seems like the very clear Gricean implicature here? Am I wrong? If not, what did you mean to say in that sentence?
All the other stuff you say seems fine. I definitely agree MIRI talked about building AIs that would be very powerful and also considered whether power concentration would be a good thing, as it would reduce race dynamics. But again, I am just talking about the part of the OP says that it was MIRI's plan to build such a system and take over the world, themselves, "in service of reducing x-risk". None of the above seems like much evidence for that? If you agree that this was not MIRI's plan, then sure, we are on the same page.
The part in square brackets seems like the very clear Gricean implicature here? Am I wrong? If not, what did you mean to say in that sentence?
See the two sentences right after.
That was at the very least, advising others to think seriously about using aligned AI to take over the world, on the basis that the world was otherwise doomed without a pivotal act. Then there was the matter of how much leverage MIRI thought they had as an organization, which is complicated by the confidentiality.
The Griecian implicature of this is that I at least don't think it's clear that MIRI wanted to build an AI to take over the world themselves. Rather, they were encouraging pivotal acts generally, and there's ambiguity about how much they were individually trying to do so.
The literal implication of this is that it's hard for people to know how much leverage MIRI has as an organization, which implies it's hard for them to know that MIRI wanted to take over the world themselves.
Cool, yeah. I mean, I can't rule this out confidently, but I do pretty strongly object to summarizing this state of affairs as:
Of course the most central old debate was over whether MIRI's plan, to build a Friendly AI to take over the world in service of reducing x-risks, was a good one.
Like, at least in my ethics there is a huge enormous gulf between trying to take over the world, and saying that it would be a good idea for someone, ideally someone with as much legitimacy as possible, who is going to build extremely powerful AI systems anyways, to do this:
- upload humans and run them at speeds more comparable to those of an AI
- prevent the origin of all hostile superintelligences (in the nice case, only temporarily and via strategies that cause only acceptable amounts of collateral damage)
- design or deploy nanotechnology such that there exists a direct route to the operators being able to do one of the other items on this list (human intelligence enhancement, prevent emergence of hostile SIs, etc.)
I go around and do the latter all the time, and think more people should do so! I agree I can't rule out from the above that MIRI was maybe also planning to build such systems themselves, but I don't currently find it that likely, and object to people referring to it as a fact of common knowledge.
I kinda thought the sales pitch for “build a Friendly AI to take over the world in service of reducing x-risks”[1] was: This is a very bad plan, but every other possible plan is even worse.
For example, the “dath ilan” stuff is IIUC what Eliezer views as an actually good approach to ASI x-risk, and it sure doesn’t look anything like that.
Anyway, if that’s the pitch, you can’t argue against it by listing reasons why the plan is very bad, right? We already know the plan is very bad. Instead you need to compare two alternatives.
(For example, if the alternative is “nobody on Earth builds ASI”, then the crux is feasibility. IIUC Eliezer’s perspective (today) is: nope that’s not feasible, the best we can hope for is to buy some time before ASI, like maybe up to an extra decade or two, and then after that we’d still need a different plan.)
Anyway, what’s your preferred alternative? I’m sure you’ve written about it somewhere but I think it belongs in this post too, to allow a side-by-side comparison.
- ^
Did you mean to write “build a Task AI to perform a pivotal act in service of reducing x-risks”? Or did MIRI switch from one to the other at some point early on? I don’t know the history. …But it doesn’t matter, my comment applies to both.
1Anyway, what’s your preferred alternative? I’m sure you’ve written about it somewhere but I think it belongs in this post too, to allow a side-by-side comparison.
Good point. It's in my first link, but I should probably put it in the current post somewhere. Here it is in the mean time:
Given that there are known ways to significantly increase the number of geniuses (i.e., von Neumann level, or IQ 180 and greater), by cloning or embryo selection, an obvious alternative Singularity strategy is to invest directly or indirectly in these technologies, and to try to mitigate existential risks (for example by attempting to delay all significant AI efforts) until they mature and bear fruit (in the form of adult genius-level FAI researchers).
- MIRI was rolling their own metaethics (deploying novel or controversial philosophy) which is not a good idea even if alignment turned out to be not that hard in a technical sense.
What specifically is this referring to? The Mere Goodness sequences?
I read your recent post about not rolling your own metaethics as addressed mostly at current AGI or safety researchers who are trying to build or align AIs today. I had thought what you were saying was that those researchers would be better served by stopping what they are doing with AI research, and instead spend their time carefully studying / thinking about / debating / writing about philosophy and metaethics. If someone asked me, I would point to Eliezer's metaethics sequences (and some of your posts and comments, among others) as a good place to start with that.
I don't think Eliezer got everything right about philosophy, morality, decision theory, etc. in 2008, but I don't know of a better / more accessible foundation, and he (and you) definitely got some important and basic ideas right, which are worth accepting and building on (as opposed to endlessly rehashing or recursively going meta on).
Is your view that it was a mistake to even try writing about metaethics while also doing technical alignment research in 2008? Or that the specific way Eliezer wrote those particular sequences is so bad / mistaken / overconfident, that it's a central example of what you want to caution against with "rolling your own metaethics"? Or merely that Eliezer did not "solve" metaethics sufficiently well, and therefore he (and others) were mistaken to move ahead and / or turn their attention elsewhere? (Either way / regardless, I still don't really know what you are concretely recommending people do instead, even after reading this thread.)
My position is a combination of:
- Eliezer was too confident in his own metaethics, and in his decision theory to a lesser degree (unlike metaethics, he never considered decision theory a solved problem, but was also willing to draw stronger practical conclusions from it than I think was justified) (and probably other philosophical positions that aren't as salient in my mind EDIT: oh yeah altruism and identity)
- Trying to solve philosophical problems like these on a deadline with intent to deploy them into AI is not a good plan, especially if you're planning to deploy it even if it's still highly controversial (i.e., a majority of professional philosophers think you are wrong). This includes Eliezer's effort as well as everyone else's.
A couple of posts arguing for 1 above:
- https://www.lesswrong.com/posts/QvYKSFmsBX3QhgQvF/morality-isn-t-logical
- https://www.lesswrong.com/posts/orhEa4wuRJHPmHFsR/six-plausible-meta-ethical-alternatives
Either way / regardless, I still don't really know what you are concretely recommending people do instead, even after reading this thread.
Did the above help you figure it out? If not, be more specific about what's confusing you about that thread?