Does it make sense to give a public response? Who would be able to do it?

The conference organizer, who had asked me to evaluate the talk, offered to interview me to set things straight. However, I don't know if that is sensible, and given my level of experience, I'm afraid I would misrepresent AI risk myself.

To be concrete: the talk was Should We Fear Intelligent Machines? by Gerald Sussman of SICP fame. He touched on important research questions and presented some interesting ideas. But much of what he said was misleading and not well-reasoned.

In response to the comments I add specifics. This is the same as I sent to the conference organizer, who had asked me for an evaluation. Note that this evaluation is separate from the interview mentioned above. The evaluation was private, the interview would be public.

  • Because of the low sound quality, I might have misunderstood some statements.

  • Mr. Sussman touched on important research questions.

  • His solution approaches might be useful.

  • He touched on some of the concerns about (strong) AI, especially the shorter term ones.

  • He acknowledged AI as a threat, which is good. But he wrongly dismissed some concerns about strong AI.

    • It's correct that current AI is not existential threat. But future AI is one.
    • He says that it won't be an existential threat, because it doesn't compete with us for resources. This is wrong.
      • Humans don't need silicon to live, but they do need silicon (and many other computer ingredients) to build much of their infrastructure. Of course we don't need that infrastructure to survive as a species. But when people talk about existential risks, they're usually not satisfied with bare survival: (section 1.2)
      • There is enough energy from the sun only if you figure out how to use it. We haven't and AI might not either in the beginning. We can't expect that it will say ‘let me be nice and leave the fossil fuels to the humans and figure out a way to use something else myself’. (Mind that I don't necessarily expect AI to be conscious like that.)
      • If AI plasters every available surface (or orbit) with solar panels, life will be dire for humans.
      • Even if it doesn't compete for resources (inputs), the outputs might be problematic. – A computer can work swimming in a lake of toxic waste at 190 °F, a human cannot.
      • (Note that I'm not assuming ‘evil AI’, but misunderstood/misaligned values. That's why it's called ‘AI alignment’.)
    • Competing with us for resources is only one way that AI is a threat. See the third section of for what people are most worried about.
    • Hawking and Musk are not the people who have thought most about AI. One needs to refute other people's arguments (FLI, Nick Bostrom, Eliezer Yudkowsky, Paul Christiano) to make a case against AI concerns.
  • Many of his arguments made big jumps.

    • He gave examples about how dumb AI is now/how shallow its understanding of the world is. These are true. But I didn't know what point he wanted to make. Then he says that there won't be any jobs left for intellectual work ‘fairly soon’, because productivity/person goes to infinity. This would require quite strong AI, which means that all the safety concerns are on the table, too.
    • The whole question about enforcement. – If AI is much more clever than we, how do we make sure it doesn't evade rule enforcement? If it has a "rule following module", how do we make sure it doesn't have subtle bugs? Free software might help here, but free software has bugs, too.
    • Also, AI might lie low and then bring everything down before we can enforce anything. This is called the treacherous turn. See also
  • It was hard to understand, but I think he made fun of Max Tegmark and Eliezer Yudkowsky who are very active in the field. At least Tegmark would laugh with anyone joking about him. [(This is my expectation given his public appearances. I don't know him personally.)] But those remarks do give the audience a wrong impression and are therefore not helpful.

  • Having such a talk at an engineering conference might be good, because it raises awareness, and there was a call to action. There is also the downside of things being misrepresented and misunderstood.

New to LessWrong?

New Answer
New Comment

1 Answers sorted by


Jun 27, 2019


I've given responses before where I go into detail about how I disagree with some public presentation on AI; the primary example is this one from January 2017, which Yvain also responded to. Generally this is done after messaging the draft to the person in question, to give them a chance to clarify or correct misunderstandings (and to be cooperative instead of blindsiding them).

I generally think it's counterproductive to 'partially engage' or to be dismissive; for example, one consequence of XiXiDu's interviews with AI experts was that some of them (that received mostly dismissive remarks in the LW comments) came away with the impression that people interested in AI risk were jerks who aren't really worth engaging with. For example, I might think someone is confused if they think climate change is more important than AI safety, but I don't think that it's useful to just tell them that they're confused or off-handedly remark that "of course AI safety is more important," since the underlying considerations (like the difference between catastrophic risks and existential risks) are actually non-obvious.

11 comments, sorted by Click to highlight new comments since: Today at 12:00 AM

I've read the slides of the underlying talk, but not listened to it. I currently don't expect to write a long response to this. My thoughts about points the talk touches on:

  • Existential risk vs. catastrophic risk. Often, there's some question about whether or not existential risks are even possible. On slide 7 and 8 Sussman identifies a lot of reasons to think that humans cause catastrophic risks (ecological destruction could possibly kill 90% of people, but seems much more difficult for it to kill 100% of people), and the distinction between the two is only important if you think about the cosmic endowment. But of course if we think AI is an existential threat, and we think humans make AI, then it is true that humans present an existential threat to ourselves. I also note here that Sussman identifies synthetic biology as possibly an existential risk, which raises the question of why an AI couldn't be a source of the existential risk presented by synthetic biology. (If an AI is built that wants to kill us, and that weapon is lying around, then we should be more concerned about AI because it has an opportunity.)
  • Accident risk vs. misuse risk. This article talks about it some, but the basic question is "will advanced AI cause problems because it did something no one wanted (accidents), or something bad people wanted (misuse)?". Most technical AI safety research is focused on accident risk, for reasons that are too long to describe here, but it's not crazy to be concerned about misuse risk, which seems to be Sussman's primary focus. I also think the sort of accident risks that we're concerned about require much deeper solutions that the normal sorts of bugs or accidents that one might imagine on hearing about this; the autonomous vehicle accident that occupies much of the talk is not a good testbed for thinking about what I think of as 'accident risk' and instead one should focus on something like the 'nearest unblocked strategy' article and related things.
  • Openness vs. closure. Open software allows for verifiability; I can know that lots of people have evaluated the decision-making of my self-driving car, rather than just Tesla's internal programming team. But also open software allows for copying and modification; the software used to enable drones that deliver packages could be repurposed to enable drones that deliver hand grenades. If we think a technology is 'dual use', in that it can both be used to make things better (like printing DNA for medical treatments) and worse (like printing DNA to create new viruses), we generally don't want those technologies to be open, and instead have carefully monitored access to dissuade improper use.
  • Solving near-term problems vs. long-term problems. Many people working on technical AI safety focus on applications with immediate uses, like the underlying math for how autonomous vehicles might play nicely with human drivers, and many people working on technical AI safety focus on research that will need to be done before we can safely deploy advanced artificial intelligence. Both of these problems seem real to me, and I wouldn't dissuade someone from working on near-term safety work (especially if the alternative is that they do capabilities work!). I think that the 'long-term' here is measured in "low numbers of decades" instead of "low numbers of centuries," and so it might be a mistake to call it 'long-term,' but the question of how to do prioritization here is actually somewhat complicated, and it seems better if we end up in a world where people working on near-term and long-term issues see each other as collaborators and allies instead of competitors for a limited supply of resources or attention.

Hi rmoehn,

I didn't look at the contents of that talk yet, but I felt uncomfortable about a specific speaker/talk being named and singled out for the target of rather hard-to-respond-to criticism (consider how you might take it if you came across a forum discussion calling your talk misleading and not well-reasoned, without going into any specifics), so I edited out those details for now.

I feel that the AI risk community should do its best to build friendly rather than hostile relationships with mainstream computer science researchers. In particular, there have been cases before where researchers looked at how their work was being discussed on LW, picked up a condescending tone, and decided that LW/AI risk people were not worth engaging with. Writing a response outlining one's disagreement to the talk (in the style of e.g. "Response to Cegłowski on superintelligence") wouldn't be a problem as it communicates engagement with the talk. But if we are referencing people's work in a manner which communicates a curt dismissal, I think we should be careful about naming specific people.

The question in general is fine, though. :)

[This comment is no longer endorsed by its author]Reply

I've added specifics. I hope this improves things. If not, feel free to edit it out.

Thanks for pointing out the problems with my question. I see now that I was wrong to combine strong language with no specifics and a concrete target. I would amend it, but then the context for the discussion would be gone.

Thanks, those specifics are great!

We briefly discussed this internally. I reverted Kaj's edit since I think we should basically never touch other user's content unless it is dealing with some real information hazards, or threatens violence or doxxes a specific individual (and probably some weird edge cases that are rare and can't easily enumerate, but which "broad PR concerns" are definitely not an instance of).

(We also sometimes edit user's content if there is some broken formatting or something in that reference class, though that feels like a different kind of thing)

Probably useful to clarify so people can understand how moderation works:

The semi-official norms around moderation tend to be "moderators have discretion to take actions without waiting for consensus, but then should report actions they took to other moderators for sanity checking." (I don't think this is formal policy but I'd personally endorse it being policy. Waiting for consensus on things often makes it impossible to take action in time sensitive situations, but checking in after the fact gets you most of the benefit)


This should be advertised in meta.

And also we edit other users' content when they give us permission to, which happens a lot.

(Note: posted after the parent was retracted.)

consider how you might take it if you came across a forum discussion calling your talk misleading and not well-reasoned, without going into any specifics

I would be grateful for the free marketing! (And entertainment—internet randos' distorted impressions of you are fascinating to read.) Certainly, it would be better for people to discuss the specifics of your work, but it's a competitive market for attention out there: vague discussion is better than none at all!

there have been cases before where researchers looked at how their work was being discussed on LW, picked up a condescending tone, and decided that LW/AI risk people were not worth engaging with

If I'm interpreting this correctly, this doesn't seem very consistent with the first paragraph? First, you seem to be saying that it's unfair to Sussman to make him the target of vague criticism ("consider how you might take it"). But then you seem to saying that it looks bad for "us" (you know, the "AI risk community", Yudkowski's robot cult, whatever you want to call it) to be making vague criticisms that will get us written off as cranks ("not worth engaging with"). But I mostly wouldn't expect both concerns to be operative in the same world—in the possible world where Sussman feels bad about being named and singled out, that means he's taking "us" seriously enough for our curt dismissal to hurt, but in the possible world where we're written off as cranks, then being named and singled out doesn't hurt.

(I'm not very confident in this analysis, but it seems important to practice trying to combat rationalization in social/political thinking??)

But I mostly wouldn't expect both concerns to be operative in the same world—in the possible world where Sussman feels bad about being named and singled out, that means he's taking "us" seriously enough for our curt dismissal to hurt, but in the possible world where we're written off as cranks, then being named and singled out doesn't hurt.

The world can change as a result of one of the concerns. At first you're taking someone seriously (or might at least be open to taking them seriously), then they say something hurtful, then you write them off to make it hurt less. Sour grapes.

Also, the reactions of people who are not being directly criticized but who respect the person being criticized are also important. Even if the target of the criticism never saw it, other people in the target's peer group may also feel disrespected and react in a similar way. (This is not speculation - I've seen various computer scientists have this reaction to writings on LW, many times.)

Moderators are discussing this with each other now. We do not have consensus on this.