Munk AI debate: confusions and possible cruxes

[-]Aryeh Englander2y3115

The more I think about this post, the more I think it captures my frustrations with a large percentage of the public discourse on AI x-risks, and not just this one debate event.

[-]Seth Herd2y118

I think so too, and I'm going to keep these points in mind when talking to risk skeptics. I wish others would too.

I couldn't bear to watch this particular debate, since the summary made it clear that little communication happened. Debates are a terrible way to arrive at the truth, since they put people into a soldier mindset. But even many conversations that don't start as debates turn into them, and investigating these cruxes could help turn them back into honest discussions in search of understanding.

[-]Ilio2y50

Strongly agree with the strong bits. (Weakly agree with some of the weak bits)

[-]Ratios2y155

"For one thing, if we use that logic, then everything distracts from everything. You could equally well say that climate change is a distraction from the obesity epidemic, and the obesity epidemic is a distraction from the January 6th attack, and so on forever. In reality, this is silly—there is more than one problem in the world! For my part, if someone tells me they’re working on nuclear disarmament, or civil society, or whatever, my immediate snap reaction is not to say “well that’s stupid, you should be working on AI x-risk instead”, rather it’s to say “Thank you for working to build a better future. Tell me more!”

Disagree with this point - cause prioritization is super important. For a radical example: imagine the government spending billions to rescue one man from Mars while neglecting much more cost-efficient causes. Bad actors use the trick of focusing on unimportant but controversial issues to keep everyone from noticing how they are being exploited routinely. Demanding sane prioritization of public attention is extremely important and valid. The problem is we as a society don't have norms and common knowledge around it (And even memes specifically against it, like whataboutism), but the fact it's not being done consistently doesn't mean that we shouldn't.

[-]Steven Byrnes2y2411

I once wrote a longer and more nuanced version that addresses this (copied from footnote 1 of my Response to Blake Richards post last year):

One could object that I’m being a bit glib here. Tradeoffs between cause areas do exist. If someone decides to donate 10% of their income to charity, and they spend it all on climate change, then they have nothing left for heart disease, and if they spend it all on heart disease, then they have nothing left for climate change. Likewise, if someone devotes their career to reducing the risk of nuclear war, then they can’t also devote their career to reducing the risk of catastrophic pandemics and vice-versa, and so on. So tradeoffs exist, and decisions have to be made. How? Well, for example, you could just try to make the world a better place in whatever way seems most immediately obvious and emotionally compelling to you. Lots of people do that, and I don’t fault them for it. But if you want to make the decision in a principled, other-centered way, then you need to dive into the field of Cause Prioritization, where you (for example) try to guess how many expected QALYs could be saved by various possible things you can do with your life / career / money, and pick one at or near the top of the list. Cause Prioritization involves (among other things) a horrific minefield of quantifying various awfully-hard-to-quantify things like “what’s my best-guess probability distribution for when AGI will arrive?”, or “exactly how many suffering chickens are equivalently bad to one suffering human?”, or “how do we weigh better governance in Spain against preventing malaria deaths?”. Well anyway, I’d be surprised if Blake has arrived at his take here via one of these difficult and fraught Cause-Prioritization-type analyses. And I note that there are people out there who do try to do Cause Prioritization, and AFAICT they very often wind up putting AGI Safety right near the top of their lists.
I wonder whether Blake’s intuitions point in a different direction than Cause Prioritization analyses because of scope neglect? As an example of what I’m referring to: suppose (for the sake of argument) that out-of-control AGI accidents have a 10% chance of causing 8 billion deaths in the next 20 years, whereas dumb AI has 100% chance of exacerbating income inequality and eroding democratic norms in the next 1 year. A scope-sensitive, risk-neutral Cause Prioritization analysis would suggest prioritizing the former, but the latter might feel intuitively more panic-inducing.
Then maybe Blake would respond: “No you nitwit, it’s not that I have scope-neglect, it’s that your hypothetical is completely bonkers. Out-of-control AGI accidents do not have a 10% chance of causing 8 billion deaths in the next 20 years; instead, they have a 1-in-a-gazillion chance of causing 8 billion deaths in the next 20 years.” And then I would respond: “Bingo! That’s the crux of our disagreement! That’s the thing we need to hash out—is it more like 10% or 1-in-a-gazillion?” And this question is unrelated to the topic of bad actors misusing dumb AI.
[For the record: The 10% figure was just an example. For my part, if you force me to pick a number, my best guess would be much higher than 10%.]

Why didn’t I put something like the above into this post? Because my sense was that this is a digression in this context—it’s not cutting to the heart of where Melanie was coming from. Like, I’m in favor of cause prioritization as much as anyone, but I didn’t have the impression that Melanie is signed onto the project of Cause Prioritization and trying in good faith to push forward that project.

I was thinking about it more last night and am now thinking it’s kinda political. In short:

Advocating for a cause is also implicitly raising the status of the group of people / movements associated with that cause, and that’s zero-sum just as much as philanthropic dollars are.

But whereas practically nobody cares about what it means for philanthropic dollars to be zero-sum (except us nerds who care about Cause Prioritization), everybody sure cares a whole lot about what it means for inter-group status competition to be zero-sum.

But that’s all kinda going on in the background, usually covered up by rationalizations, I think. Not sure what to do about it. :(

[-]dr_s2y1310

I find arguing about timelines also frustrating for the reason that it has a huge asymmetry.

Suppose you have a distribution of probability for times at which AGI might appear, which is the best you can really honestly say to have (if you claim to have a single precise date, either you're a prophet or you don't know what you're talking about).

The AI safety argument only needs to demonstrate that the total P(T < 20 years) is higher than, say, 5% to argue that AI safety isn't just a problem, it's a relatively urgent problem.

The AI optimism argument needs to argue that P(T < 20 years) or even P(T < 50 years) is essentially zero, because otherwise the consequences are potentially so massive it really takes very little to offset even a low probability. How do you do that? I understand thinking that AGI is probably kinda far off. How do you get absolutely sure that AGI can't possibly be behind the corner? I mean willing-to-bet-and-be-shot-in-the-head-if-you-lose sure? Because that's what we're talking about here. I don't see how any intellectually honest technologist can be that confident about such a hard to predict thing.

There's very few things I'd be willing to bet my life on. Specific-ish dates for conceivably feasible technological developments are not one of them. And if you do accept that AGI is possible, and you do think that it's powerful and in fact worth seeking, and you justify your work by saying it will revolutionize the world, but you also deny that it can possibly be correspondingly dangerous, then you're just not a serious person.

[-]Seth Herd2y711

I think this is a critical point in favor of AGI safety concern. Distilling this argument to a talking point seems useful for the public debate.

[-]Roman Leventov2y21

"Willing-to-bet-and-be-shot-in-the-head-if-you-lose" starts to be a little too much Pascal's Mugging. By this token we can make different arguments about how unimaginable high suffering (S-risk) happens right now, which seems plausible enough that nobody would be willing to put their life on a bet about it, and the implications of these arguments might be in facts that AI should mostly wipe out the complex life on earth and start from scratch. I.e.., this would motivate that consciousness and ethics research is even higher, or at least equal priority with more "orthodox" style of AI x-risk concen, which takes that human's or at least complex life's existence on earth is an unshackable assumption (prior).

But in general, I agree with the sentiment of your comment.

[-]dr_s2y10

I say it just to mean that the level of certainty required is quite high. Even if you count only human deaths and not all the rest of the value lost, with 8 billion people, a 0.0001% chance of extinction is an expected value of 8,000 deaths. People would usually be quite careful about boldly stating something that can get 8,000 people killed! Anyone who's trying to argue that "no worries, it'll be fine" has a much higher burden of proof for that reason alone, IMO, especially if they want to essentially dodge altogether the argument for why it might not, and simply rely on "there's no way we'll invent AGI that soon anyway", which many people do.

[-]M. Y. Zuo2y60

Thanks for the interesting writeup.

It seemed to me that all four participants (and the moderator!) were making timelines and LLM-related arguments, in ways that were both annoyingly vague, and unrelated to the statement under debate.
(If astronomers found a giant meteor projected to hit the earth in the year 2123, nobody would question the use of the term “existential threat”, right??)

The giant meteor likely wouldn't be questioned in that way if multiple independent groups (i.e. NASA, Roscosmos, ESA, CNSA, etc...) announced it with a decent probability, (maybe at least a 5% to 10% chance?) and publicized their calculations.

It's a lot more difficult for LLM timeline arguments because there are no existing guaranteed-to-be-independent-from-each-other groups with a comparable level of reputation and credibility to stake.

[-]Roman Leventov2y10

Even if something precluded astronomers from making principled calculations of the impact probability up until five years before the date of the hit, and before that the maximum that they could say would be "we cannot confidently rule out the possibility of an impact", this would still be treated like an existential threat.

[-]M. Y. Zuo2y10

It doesn't seem like anything could preclude all of the world's astronomers from making calculations regarding impact events.

Can you explain how the presence or absence of such an unlikely restriction relates to your latter point?

[-]Ilio2y50

Great post, even from my not-necessarily-disagreeing position. It helps me consolidate the update that it’s not necessarily useless to start preparing in case another unexpected revolution hits.

It feels like it’s accelerating, right?

One disagreement I think we should be able to clear is: should we evaluate p(x-risk caused by AIs) in isolation? or should we normalize by p(x-risk avoided by AIs)?

Fun fact: I happen to suspect you might be the one LW regular closer to designing a true AGI, so maybe it’s good you play safe. 😉

[-]Steven Byrnes2y52

Thanks!

I think we should try to form correct beliefs about p(x-risk caused by AIs), and we should also try to form correct beliefs about p(x-risk avoided by AIs), and then we should make sensible decisions in light of those beliefs. I don’t see any reason to combine those two probabilities into a single “yay AI versus boo AI” axis—see Section 3 above! :)

For example, if p(x-risk caused by AIs) is high, and p(x-risk avoided by AIs) is even higher, then we should brainstorm ways to lower AI x-risk other than “stop doing AI altogether forever” (as if that were a feasible option in the first place!); and we should also talk about what those other x-risks are, and whether there are non-AI ways to mitigate them; and we should also talk about how both those probabilities might change if we somehow make AGI happen N years later (or N years sooner), etc. Whereas if p(x-risk caused by AIs) is ≈0, we would be asking different questions and facing different tradeoffs. Right?

[-]Ilio2y*10

[on second thought some material removed for better usage]

I think we should try to form correct beliefs about p(x-risk caused by AIs), and we should also try to form correct beliefs about p(x-risk avoided by AIs), and then we should make sensible decisions in light of those beliefs.

Hear! Hear!

Whereas if p(x-risk caused by AIs) is ≈0, we would be asking different questions and facing different tradeoffs. Right?

I don’t know. It doesn’t feel like that’s why I choose to spend some time checking these questions in the first place.

You know what it feels like when your plane is accelerating and you start feel like something is going to happen, then you remember: of course, it’s a take off.

Well, Imagine you still feel acceleration after the take off. First, you guess it’s to hard to evaluate the distance but you know it must stop when you’re at cruising speed. Right? After one hour, you keep feeling acceleration, but that must be a mistake because a Fermi estimate says you can’t be accelerating at this point. Right? Then it accelerates more. Right? This is how I feel after I found out about convolution, relu, alphago,, alphazero. « Attention is all you need » and children.

[-][anonymous]2y518

Good post, very well written!

[-]Abhinav Srivastava2y10

(If astronomers found a giant meteor projected to hit the earth in the year 2123, nobody would question the use of the term “existential threat”, right??)

I'm wondering if you believe that AGI risk is equivalent to a giant meteor hitting the Earth or was that just a throw away analogy? This helps me get a better idea of where x-risk concern-havers (?) stand on the urgency of the risk. /gen

Thanks

[-]Steven Byrnes2y20

I was just making a narrow point: the term “existential threat” does not generally have a connotation of “imminent”.

An analysis in The Precipice concludes that the risk of extinction via “natural” asteroid or comet impact is around 1 in 1,000,000 in the next century. I think the probability of human extinction via AI in the next century is much much much much higher than 1 in 1,000,000. If you force me to pick a number, I would say something above 50% in the next 30 years. That’s just my opinion though. There’s quite a range of opinions in the field. I generally think it’s pretty hard to say, within a pretty broad range, at least in my present state of knowledge. However, if someone says it’s below 1% in the next century, then I feel very strongly that they have not thought it through sufficiently carefully, and are overlooking important considerations.

[-]Jonah Wilberg2y10

Very useful post, thanks. While the 'talking past each other' is frustrating, the 'not necessarily disagreeing' suggests the possibility of establishing surprising areas of consensus. And it might be interesting to explore further what exactly that consensus is. For example:

Yann suggested that there was no existential risk because we will solve it

I'm sure the air of paradox here (because you can't solve a problem that doesn't exist) is intentional, but if we drill down, should we conclude that Yann actually agrees that there is an existential risk (just that the probabilities are lower than other estimates, and less worth worrying about)? Yann sometimes compares the situation to the risk of designing a car without brakes, but if the car is big enough that crashing it would destroy civilization that still kinda sounds like an existential risk.

I'm also not sure the fire department analogy helps here - as you note later in the paper Yann thinks he knows in outline how to solve the problem and 'put out the fires', so it's not an exogenous view. It seems like the difference between the fire chief who thinks their job is easy vs the one who thinks it's hard, though everyone agrees fires spreading would be a big problem.

[-]Steven Byrnes2y41

Yeah, like I said, I don’t think that one in particular was a major dynamic, just one thing I thought worth mentioning. I think one could rephrase what they said slightly to get a very similar disagreement minus the talking-past-each-other.

Like, for example, if every human jumped off a cliff simultaneously, that would cause extinction. Is that an “x-risk”? No, because it’s never going to happen. We don’t need any “let’s not all simultaneously jump off a cliff” activist movements, or any “let’s not all simultaneously jump off a cliff” laws, or any “let’s not all simultaneously jump off a cliff” fields of technical research, or anything like that.

That’s obviously a parody, but Yann is kinda in that direction regarding AI. I think his perspective is: We don’t need activists, we don’t need laws, we don’t need research. Without any of those things, AI extinction is still not going to happen, just because that’s the natural consequence of normal human behavior and institutions doing normal stuff that they’ve always done.

I think “Yann thinks he knows in outline how to solve the problem” is maybe giving the wrong impression here. I think he thinks the alignment problem is just a really easy problem with a really obvious solution. I don’t think he’s giving himself any credit. I think he thinks anyone looking at the source code for a future human-level AI would be equally capable of making it subservient with just a moment’s thought. His paper didn’t really say “here’s my brilliant plan for how to solve the alignment problem”, the vibe was more like “oh and by the way you should obviously choose a cost function that makes your AI kind and subservient” as a side-comment sentence or two. (Details here)

[-]Noosphere892y*2-2

Re how hard alignment is, I suspect it's harder on my median world than Yann LeCun thinks, but probably a lot easier than most LWers think, and in particular I assign some probability mass to the hypothesis that in practice, AI alignment of superhuman AI is a non-problem.

This means that while I disagree with Yann LeCun, and in general find my side (anti-doomerism/accelerationism) to be quite epistemically unsound at best, I do suspect that there are object level reasons to believe the alignment of superhuman systems is way easier than most LWers think, including you.

To give the short version, instrumental convergence is probably going to be constrained for pure capabilities reasons, and in particular unconstrained instrumental goals via RL usually fail, or at best produce something useless. This means that it will be reasonably easy to add constraints to instrumental goals, and the usual thought experiment of a squiggle maximizer basically collapses, because they won't have the capabilities. This implies that we are dealing with an easier problem, since we aren't starting from zero, and in the strongest form, basically transforms it into a non-adversarial problem like nuclear safety, which we solve easily and rather well.

Or in slogan form "Less constraints on instrumental goals is usually not good for capabilities, and the human case is probably a result of just not caring about efficiency, plus time scales. We should expect AI systems to have more constraints on instrumental goals for capability and alignment reasons."

^{^}

Note: For the purpose of this post, I’m mostly assuming that all four debaters were by-and-large saying things that they believe, in good faith, as opposed to trying to win the debate at all costs. Call me naïve.

^{^}

Hmm, the best I can think of (in this hypothetical) is that we could note that “Earth species have sometimes wiped each other out”. And we could try to make a mathematical model based on that. And we could use the past history of extinctions as empirical data to validate that model. But the extraterrestrial UFO fleet is pretty different! We’re on shaky grounds applying that same model in such a different situation.

And anyway, if someone makes a quantitative model based on “Earth species have sometimes wiped each other out”, and if that counts as “empirically validated” justification for calling the incoming alien fleet an x-risk, then I claim that very same quantitative model should count as “empirically validated” justification for calling future advanced AI an x-risk too! After all, future advanced AI will probably have a lot in common with a new intelligent nonhuman species, I claim.

LESSWRONG
LW

LESSWRONG
LW

244

Munk AI debate: confusions and possible cruxes

244

244

Five ways people were talking past each other

1. Treating efforts to solve the problem as exogenous or not

2. Ambiguously changing the subject to “timelines to x-risk-level AI”, or to “whether large language models (LLMs) will scale to x-risk-level AI”

3. Vibes-based “meaningless arguments”

4. Ambiguously changing the subject to policy

5. Ambiguously changing the subject to Cause Prioritization

Some possible cruxes of why I and Yann / Melanie disagree about x-risk

Yann LeCun

Melanie Mitchell

A. “The field of AI is more than 30 years away from X” is not the kind of claim that one should make with 99%+ confidence

B. …And 30 years is still part of the “foreseeable future”

C. “Common sense” has an ought-versus-is split, and it’s possible to get the latter without the former

D. It is conceivable for something to be an x-risk without there being any nice clean quantitative empirically-validated mathematical model proving that it is