I think it raises some very important points. OTTMH, I think the most important one is: We have no good critics.There is nobody I'm aware of who is seriously invested in knocking down AI-Xrisk arguments and qualified to do so. For many critics in machine learning (like Andrew Ng and Yann Lecun), the arguments seem obviously wrong or misguided, and so they do not think it's worth their time to engage beyond stating that.
A related point which is also important is: We need to clarify and strengthen the case for AI-Xrisk.Personally, I think I have a very good internal map of the path arguments about AI-Xrisk can take, and the type of objections one encounters. It would be good to have this as some form of flow-chart. Let me know if you're interested in helping make one.
Regarding machine learning, I think he made some very good points about how the the way ML works doesn't fit with the paperclip story. I think it's worth exploring the disanalogies more and seeing how that affects various Xrisk arguments.
As I reflect on what's missing from the conversation, I always feel the need to make sure it hasn't been covered in Superintelligence. When I read it several years ago, I found Superintelligence to be remarkably thorough. For example, I'd like to point out that FOOM isn't necessary for a unilateral AI-takeover, since an AI could be progressing gradually in a box, and then break out of the box already superintelligent; I don't remember if Bostrom discussed that.
The point about justification drift is quite apt. For instance, I think the case for MIRI's veiwpoint increasingly relies on:
1) optimization daemons (aka "inner optimizers")
2) adversarial examples (i.e. current ML systems seem to learn superficially similar but deeply flawed versions of our concepts)
TBC, I think these are quite good arguments, and I personally feel like I've come to appreciate them much more as well over the last several years. But I consider them far from conclusive, due to our current lack of knowledge/understanding.
One thing I didn't quite agree with in the talk: I think he makes a fairly general case against trying to impact the far future. I think the magnitude of impact and uncertainty we have about the direction of impact mostly cancel each other out, so even if we are highly uncertain about what effects our actions will have, it's often still worth making guesses and using them to inform our decisions. He basically acknowledges this.
Re: your request for collaboration - I am skeptical of ROI of research on AI X-risk, and I would be happy to help offer insight on that perspective, either as a source or as a giver of feedback. Feel free to email me at {last name}{first name}@gmail.com
I'm not an expert in AI, but I have a PhD in semiconductors (which gives me perspective on hardware) and currently work on machine learning at Netflix (which gives me perspective on software). I also was one of the winners of the SciCast prediction market a few years back, which is evidence that my judgment of near-term tech trends is decently calibrated.
As someone who sometimes writes things that are a bit skeptical regarding AI doom, I find the difficulty of getting counter-criticism frustrating.
11Raemon
Nod. That does seem like another important part of the equation. I admit I have not read that much of your work. I just went to revisit the last essay of yours I remember looking at and amusingly found this comment of mine apologizing for not having commented more. :(
I liked this talk by Ben.
I think it raises some very important points. OTTMH, I think the most important one is: We have no good critics. There is nobody I'm aware of who is seriously invested in knocking down AI-Xrisk arguments and qualified to do so. For many critics in machine learning (like Andrew Ng and Yann Lecun), the arguments seem obviously wrong or misguided, and so they do not think it's worth their time to engage beyond stating that.
A related point which is also important is: We need to clarify and strengthen the case for AI-Xrisk. Personally, I think I have a very good internal map of the path arguments about AI-Xrisk can take, and the type of objections one encounters. It would be good to have this as some form of flow-chart. Let me know if you're interested in helping make one.
Regarding machine learning, I think he made some very good points about how the the way ML works doesn't fit with the paperclip story. I think it's worth exploring the disanalogies more and seeing how that affects various Xrisk arguments.
As I reflect on what's missing from the conversation, I always feel the need to make sure it hasn't been covered in Superintelligence. When I read it several years ago, I found Superintelligence to be remarkably thorough. For example, I'd like to point out that FOOM isn't necessary for a unilateral AI-takeover, since an AI could be progressing gradually in a box, and then break out of the box already superintelligent; I don't remember if Bostrom discussed that.
The point about justification drift is quite apt. For instance, I think the case for MIRI's veiwpoint increasingly relies on:
1) optimization daemons (aka "inner optimizers")
2) adversarial examples (i.e. current ML systems seem to learn superficially similar but deeply flawed versions of our concepts)
TBC, I think these are quite good arguments, and I personally feel like I've come to appreciate them much more as well over the last several years. But I consider them far from conclusive, due to our current lack of knowledge/understanding.
One thing I didn't quite agree with in the talk: I think he makes a fairly general case against trying to impact the far future. I think the magnitude of impact and uncertainty we have about the direction of impact mostly cancel each other out, so even if we are highly uncertain about what effects our actions will have, it's often still worth making guesses and using them to inform our decisions. He basically acknowledges this.