Hi Jessica. Nice post and I agree with many of your points. Certainly, I believe—as you do—that a number of bad actors are wielding the specter of AGI sloppily and irresponsibly, either to consciously defraud people or on account of buying into something that speaks more to the messianic than to science. Perhaps ironically, one frequent debate that I have had with Gary in the past is that while he is vocally critical of exuberance over deep learning, he is himself partial to speaking rosily of nearish-term AGI, and of claiming progress (or being on the verge of progress) towards it. On the other hand, I am considerably more skeptical.
While I enjoyed the post and think we agree on many points, if you don’t mind I would like to respectfully note that I’ve been quoted here slightly out of context and would like to supply that missing context. To be sure, I think your post is written well and with honest intentions, and I know how easy it is to miss some context in Twitter threads [especially as it seems that many tweets have been deleted from this thread].
Regarding my friend Gary Marcus. I like Gary and we communicate fairly regularly, but we don’t always agree on the science or on the discourse.
In this particular case, he was specifically insisting to be “the first” to make a set of arguments (which contradicted my understanding of history). When I say “Saying it louder ≠ saying it first.”, I am truly just pushing back on this specific point—the assertion that he said it first. Among others, Judea Pearl has argued the limits of curve fitting far more rigorously and far earlier in history.
There is nothing dishonest or contradictory about agreeing with a broader point and simultaneously disagreeing with a man’s claim to have originated it. I took exception to the hyperbole, not the message.
After the quote, your post notes: “but, this is essentially admitting that Marcus is correct, while still criticizing him for saying it”—what is there to “admit”? I myself have made similar critical arguments both in technical papers, position papers, blog posts, and the popular press. Characterizing “agreement” as “admitting” something, makes the false insinuation that somehow I have been on the wrong side of the debate.
For the record, I am acknowledged in Gary’s paper on limitations of deep learning (which you reference here) for giving him a large amount of constructive feedback and have myself, perhaps in defiance of Adam Smith’s aphorism, been vocally critical of my own community within the technical forums, recently publishing “Troubling Trends in Machine Learning Scholarship” at the ICML debates (https://arxiv.org/abs/1807.03341) which was subsequently published by CACM. I suspect this piece (as well as much of my writing at http://approximatelycorrect.com and in other formal position papers) is in the spirit of the sort of critical writing that you appear to encourage.
Further, I’d like to address the second part of the discussion. When I say “Saying X doesn’t solve Y is pretty easy. But where are your concrete solutions for Y?” My point here is that Gary doesn’t just push back on the false claims made about current technology that doesn’t do Y. He also sometimes makes public attacks on the people working on X. It would seem that their crime is that they haven’t developed technical solutions to the grand challenge of Y. If failing to solve these particular moonshots (true reasoning, solving common sense, an elegant formulation of symbol manipulation and synthesis with pattern recognition) is a crime, then Gary too is just as guilty and the attack ought to be levied with greater humility. These attacks strike me as inappropriate and misplaced (compared to the more reasonable push-back on misinformation in the public sphere. ***To be clear, while I understand why you might have drawn that conclusion from this half-tweet, I do not believe that one must have a solution in hand to levy criticism and my writing and technical papers attest to this.***
[epistemic status: that's just my opinion, man. I have highly suggestive evidence, not deductive proof, for a belief I sincerely hold]
"If you see fraud and do not say fraud, you are a fraud." --- Nasim Taleb
I was talking with a colleague the other day about an AI organization that claims:
His response was (paraphrasing): "Wow, that's a really good lie! A lie that can't be disproven."
I found this response refreshing, because he immediately jumped to the most likely conclusion.
Near predictions generate more funding
Generally, entrepreneurs who are optimistic about their project get more funding than ones who aren't. AI is no exception. For a recent example, see the Human Brain Project. The founder, Henry Makram, predicted in 2009 that the project would succeed in simulating a human brain by 2019, and the project was already widely considered a failure by 2013. (See his TED talk, at 14:22)
The Human Brain project got 1.3 billion Euros of funding from the EU.
It's not hard to see why this is. To justify receiving large amounts of money, the leader must make a claim that the project is actually worth that much. And, AI projects are more impactful if it is, in fact, possible to develop AI soon. So, there is an economic pressure towards inflating estimates of the chance AI will be developed soon.
Fear of an AI gap
The missile gap was a lie by the US Air Force to justify building more nukes, by falsely claiming that the Soviet Union had more nukes than the US.
Similarly, there's historical precedent for an AI gap lie used to justify more AI development. Fifth Generation Computer Systems was an ambitious 1982 project by the Japanese government (funded for $400 million in 1992, or $730 million in 2019 dollars) to create artificial intelligence through massively parallel logic programming.
The project is widely considered to have failed. From a 1992 New York Times article:
(emphasis mine)
This bears similarity to some conversations on AI risk I've been party to in the past few years. The fear is that Others (DeepMind, China, whoever) will develop AGI soon, so We have to develop AGI first in order to make sure it's safe, because Others won't make sure it's safe and We will. Also, We have to discuss AGI strategy in private (and avoid public discussion), so Others don't get the wrong ideas. (Generally, these claims have little empirical/rational backing to them; they're based on scary stories, not historically validated threat models)
The claim that others will develop weapons and kill us with them by default implies a moral claim to resources, and a moral claim to be justified in making weapons in response. Such claims, if exaggerated, justify claiming more resources and making more weapons. And they weaken a community's actual ability to track and respond to real threats (as in The Boy Who Cried Wolf).
How does the AI field treat its critics?
Hubert Dreyfus, probably the most famous historical AI critic, published "Alchemy and Artificial Intelligence" in 1965, which argued that the techniques popular at the time were insufficient for AGI. Subsequently, he was shunned by other AI researchers:
This makes sense as anti-whistleblower activity: ostracizing, discrediting, or punishing people who break the conspiracy to the public. Does this still happen in the AI field today?
Gary Marcus is a more recent AI researcher and critic. In 2012, he wrote:
In 2018, he tweeted an article in which Yoshua Bengio (a deep learning pioneer) seemed to agree with these previous opinions. This tweet received a number of mostly-critical replies. Here's one, by AI professor Zachary Lipton:
The first criticism is essentially a claim that everybody knows that deep learning can't do reasoning. But, this is essentially admitting that Marcus is correct, while still criticizing him for saying it [ED NOTE: the phrasing of this sentence is off (Lipton publicly agrees with Marcus on this point), and there is more context, see Lipton's reply].
The second is a claim that Marcus shouldn't criticize if he doesn't have a solution in hand. This policy deterministically results in the short AI timelines narrative being maintained: to criticize the current narrative, you must present your own solution, which constitutes another narrative for why AI might come soon.
Deep learning pioneer Yann LeCun's response is similar:
Again, the criticism is not that Marcus is wrong in saying deep learning can't do certain forms of reasoning, the criticism is that he isn't presenting an alternative solution. (Of course, the claim could be correct even if Marcus doesn't have an alternative!)
Apparently, it's considered bad practice in AI to criticize a proposal for making AGI without presenting on alternative solution. Clearly, such a policy causes large distortions!
Here's another response, by Steven Hansen (a research scientist at DeepMind):
That is: to criticize the field, you should go through the field, not through the press. This is standard guild behavior. In the words of Adam Smith: "People of the same trade seldom meet together, even for merriment and diversion, but the conversation ends in a conspiracy against the public, or in some contrivance to raise prices."
(Also see Marcus's medium article on the Twitter thread, and on the limitations of deep learning)
[ED NOTE: I'm not saying these critics on Twitter are publicly promoting short AI timelines narratives (in fact, some are promoting the opposite), I'm saying that the norms by which they criticize Marcus result in short AI timelines narratives being maintained.]
Why model sociopolitical dynamics?
This post has focused on sociopolotical phenomena involved in the short AI timelines phenomenon. For this, I anticipate criticism along the lines of "why not just model the technical arguments, rather than the credibility of the people involved?" To which I pre-emptively reply:
Which is not to say that modeling such technical arguments is not important for forecasting AGI. I certainly could have written a post evaluating such arguments, and I decided to write this post instead, in part because I don't have much to say on this issue that Gary Marcus hasn't already said. (Of course, I'd have written a substantially different post, or none at all, if I believed the technical arguments that AGI is likely to come soon had merit to them)
What I'm not saying
I'm not saying:
I'm saying that there are systematic sociopolitical phenomena that cause distortions in AI estimates, especially towards shorter timelines. I'm saying that people are being duped into believing a lie. And at the point where 73% of tech executives say they believe AGI will be developed in the next 10 years, it's a major one.
This has happened before. And, in all likelihood, this will happen again.