A case for courage, when speaking of AI danger

So8res

While I disagree with Nate on a wide variety of topics (including implicit claims in this post), I do want to explicitly highlight strong agreement with this:

I have a whole spiel about how your conversation-partner will react very differently if you share your concerns while feeling ashamed about them versus if you share your concerns as if they’re obvious and sensible, because humans are very good at picking up on your social cues. If you act as if it’s shameful to believe AI will kill us all, people are more prone to treat you that way. If you act as if it’s an obvious serious threat, they’re more likely to take it seriously too.

The position that is "obvious and sensible" doesn't have to be "if anyone builds it, everyone dies". I don't believe that position. It could instead be "there is a real threat model for existential risk, and it is important that society does more to address it than it is currently doing". If you're going to share concerns at all, figure out the position you do have courage in, and then discuss that as if it is obvious and sensible, not as if you are ashamed of it.

(Note that I am not convinced that you should always be sharing your concerns. This is a claim about how you should share concerns, conditional on having decided that you are going to share them.)

[-][anonymous]18d2514

It could instead be "there is a real threat model for existential risk, and it is important that society does more to address it than it is currently doing". If you're going to share concerns at all, figure out the position you do have courage in, and then discuss that as if it is obvious and sensible, not as if you are ashamed of it.

The potential failure mode I see with this is that, if you're not paying sufficient attention to your rhetoric, you run the risk of activating people's anti-Pascal's mugging instincts.^[1] As Jeremy Gillen said:

Everyone is pretty frequently bombarded with wild arguments and beliefs that have crazy implications. Like conspiracy theories, political claims, spiritual claims, get-rich-quick schemes, scientific discoveries, news headlines, mental health and wellness claims, alternative medicine, claims about which lifestyles are better. We don't often have the time (nor expertise or skill or sometimes intelligence) to evaluate them properly.

"You should pay attention to this even if you think there's a really small chance of it happening because if it does happen, the consequences will be massive" is something most people hear pretty regularly. In lieu o... (read more)

[-]So8res17d*5135

I don't think most anyone who's studied the issues at hand thinks the chance of danger is "really small", even among people who disagree with me quite a lot (see e.g. here). I think folks who retreat to arguments like "you should pay attention to this even if you think there's a really small chance of it happening" are doing a bunch of damage, and this is one of many problems I attribute to a lack of this "courage" stuff I'm trying to describe.

When I speak of "finding a position you have courage in", I do not mean "find a position that you think should be logically unassailable." I'm apparently not doing a very good job at transmitting the concept, but here's some positive and negative examples:

✓ "The race towards superintelligence is ridiculously risky and I don't think humanity should be doing it."
✓ "I'm not talking about a tiny risk. On my model, this is your most likely cause of death."
✓ "Oh I think nobody should be allowed to race towards superintelligence, but I'm trying to build it anyway because I think I'll do it better than the next guy. Ideally all AI companies in the world should be shut down, though, because we'd need way more time to do this properly." (The action i... (read more)

[-][anonymous]17d10-5

I don't think most anyone who's studied the issues at hand thinks the chance of danger is "really small", even among people who disagree with me quite a lot (see e.g. here)

This link doesn't seem to include people like Quintin Pope and the AI Optimists, who are the most notorious AI risk skeptics I can think of who have nonetheless written about Eliezer's arguments (example). If I recall correctly, I think Pope said sometime before his departure from this site that his P(doom) is around 1%.

9TurnTrout16d

Yup, that claim is wrong. I'm not <= 1% but I have met educated skeptics who are. Not sure why Nate made this claim since it isn't relevant to his point -- could just delete that first sentence.

[-]So8res16d1717

(The existence of exceptions is why I said "most anyone" instead of "anyone".)

9SAB257d

Something I notice is that in the good examples you use only I statements. "I don't think humanity should be doing it", "I'm not talking about a tiny risk", "Oh I think I'll do it better than the next guy". Whereas in the bad examples it's different, "Well we can all agree that it'd be bad if AIs were used to enable terrorists to make bioweapons", "Even if you think the chance of it happening is very small", "In some unlikely but extreme cases, these companies put civilization at risk" I think with the bad examples there's a lot of pressure for the other person to agree, "the companies should be responsible (because I say so)", "Even if you think... Its still worth focusing on (because I've decided what you should care about)", "Well we can all agree (I've already decided you agree and you're not getting a choice otherwise)" Whereas with the good examples the other person is not under any pressure to agree, so they are completely free to think about the things you're saying. I think that's also part of what makes these statements courageous, that it's stated in a way where the other person is free to agree or dissagree as they wish, and so you trust that what your saying is compelling enough to be persuasive on its own.

6Viliam16d

I have a similar experience from reading Hacker News. Seems to me that people who write it don't really want to express an opinion on AI, more like they are using absurdity heuristic for an attack by association against EA. (Attacking EA is their goal, arguing by "AI apocalypse is unlikely" is merely a convenient weapon.)

[-]Buck19d8245

Both Soares and me get mixed reviews for our social presentation, so you might want to take all this with a grain of salt. But here's my two cents of response.

I agree it's good to not seem ashamed of something you're saying. I think this is mostly a special case of how it's good to be personally confident. (See Soares here for some good advice on what it means to be confident in a confusing situation.) One reason is that it's really helpful to convey to your conversational partner that, despite the fact you're interested in what they think, you'll be fine and chill regardless of how they respond to you; this lets them relax and e.g. say their true objections if they want to.

But I think it's generally a mistake to act as if beliefs are "obvious" if in fact they aren't obvious to your audience. That is, I think that you should talk differently when saying the following different types of statements:

Statements that your audience already agrees with
Statements that your audience already disagrees with
Statements that your audience hasn't thought about yet, but that they'll immediately agree with upon consideration
Statements that your audience hasn't thought about yet, but that they'll no

... (read more)

[-]So8res19d5330

I agree that it's usually helpful and kind to model your conversation-partner's belief-state (and act accordingly).

And for the avoidance of doubt: I am not advocating that anyone pretend they think something is obvious when they in fact do not.

By "share your concerns as if they’re obvious and sensible", I was primarily attempting to communicate something more like: I think it's easy for LessWrong locals to get lost in arguments like whether AI might go fine because we're all in a simulation anyway, or confused by turf wars about whether AI has a 90+% chance of killing us or "only" a ~25% chance. If someone leans towards the 90+% model and gets asked their thoughts on AI, I think it's worse for them to answer in a fashion that's all wobbly and uncertain because they don't want to be seen as overconfident against the 25%ers, and better for them to connect back to the part where this whole situation (where companies are trying to build machine superintelligence with very little idea of what they're doing) is wacky and insane and reckless, and speak from there.

I don't think one should condescend about the obviousness of it. I do think that this community is generally dramatically faili... (read more)

[-]Buck19d19-18

I do think that this community is generally dramatically failing to make the argument "humanity is building machine superintelligence while having very little idea of what it's doing, and that's just pretty crazy on its face" because it keeps getting lost in the weeds (or in local turf wars).

I don't think the weed/local turf wars really cause the problems here, why do you think that?

The weeds/local turf wars seem like way smaller problems for AI-safety-concerned people communicating that the situation seems crazy than e.g. the fact that a bunch of the AI safety people work at AI companies.

And I was secondarily attempting to communicate something like: I think our friends in the policy world tend to cede far too much social ground.

Idk, seems plausible.

[-]So8res19d4629

I don't think the weed/local turf wars really cause the problems here, why do you think that?

The hypothesized effect is: people who have been engaged in the weeds/turf wars think of themselves as "uncertain" (between e.g. the 25%ers and the 90+%ers) and forget that they're actually quite confident about some proposition like "this whole situation is reckless and crazy and Earth would be way better off if we stopped". And then there's a disconnect where (e.g.) an elected official asks a local how bad things look, and they answer while mentally inhabiting the uncertain position ("well I'm not sure whether it's 25%ish or 90%ish risk"), and all they manage to communicate is a bunch of wishy-washy uncertainty. And (on this theory) they'd do a bunch better if they set aside all the local disagreements and went back to the prima-facie "obvious" recklessness/insanity of the situation and tried to communicate about that first. (It is, I think, usually the most significant part to communicate!)

8Buck19d

Yeah I am pretty skeptical that this is a big effect—I don't know anyone who I think speaks unconfidently without the courage of their convictions when talking to audiences like elected officials for this kind of reason—but idk.

[-]Richard_Ngo16d3220

Whoa, this seems very implausible to me. Speaking with the courage of one's convictions in situations which feel high-stakes is an extremely high bar, and I know of few people who I'd describe as consistently doing this.

If you don't know anyone who isn't in this category, consider whether your standards for this are far too low.

9Lukas Finnveden14d

I read Buck's comment as consistent with him knowing people who speak without the courage of their convictions for other reasons than stuff like "being uncertain between 25% doom and 90% doom".

[-]So8res19d*1713

Huh! I've been in various conversations with elected officials and have had the sense that most people speak without the courage of their convictions (which is not quite the same thing as "confidence", but which is more what the post is about, and which is the property I'm more interested in discussing in this comment section, and one factor of the lack of courage is broadcasting uncertainty about things like "25% vs 90+%" when they could instead be broadcasting confidence about "this is ridiculous and should stop"). In my experience, it's common to the point of others expressing explicit surprise when someone does and it works (as per the anecdote in the post).

I am uncertain to what degree we're seeing very different conversations, versus to what degree I just haven't communicated the phenomena I'm talking about, versus to what degree we're making different inferences from similar observations.

[-]Buck18d11-3

I don't think your anecdote supports that it's important to have the courage of your convictions when talking. I think the people I know who worked on SB-1047 are totally happy to say "it's ridiculous that these companies don't have any of the types of constraints that might help mitigate extreme risks from their work" without wavering because of the 25%-vs-90% thing. I interpret your anecdote as being evidence about which AI-concerned-beliefs go over well, not about how you should say them. (Idk how important this is, np if you don't want to engage further.)

[-]So8res18d*4928

A few claims from the post (made at varying levels of explicitness) are:

1 . Often people are themselves motivated by concern X (ex: "the race to superintelligence is reckless and highly dangerous") and decide to talk about concern Y instead (ex: "AI-enabled biorisks"), perhaps because they think it is more palatable.

2 . Focusing on the "palatable" concerns is a pretty grave mistake.

2a. The claims Y are often not in fact more palatable; people are often pretty willing to talk about the concerns that actually motivate you.

2b. When people try talking about the concerns that actually motivate them while loudly signalling that they think their ideas are shameful and weird, this is not a terribly good test of claim (2a).

2c. Talking about claims other than the key ones comes at a steep opportunity cost.

2d. Talking about claims other than the key ones risks confusing people who are trying to make sense of the situation.

2e. Talking about claims other than the key ones risks making enemies of allies (when those would-be allies agree about the high-stakes issues and disagree about how to treat the mild stuff).

2f. Talking about claims other than the key ones triggers people's bullshit detector... (read more)

[-]Buck17d130

Ok. I don't think your original post is clear about which of these many different theses it has, or which points it thinks are evidence for other points, or how strongly you think any of them.

I don't know how to understand your thesis other than "in politics you should always pitch people by saying how the issue looks to you, Overton window or personalized persuasion style be damned". I think the strong version of this claim is obviously false. Though maybe it's good advice for you (because it matches your personality profile) and perhaps it's good advice for many/most of the people we know.

I think that making SB-1047 more restrictive would have made it less likely to pass, because it would have made it easier to attack and fewer people would agree that it's a step in the right direction. I don't understand who you think would have flipped from negative to positive on the bill based on it being stronger—surely not the AI companies and VCs who lobbyied against it and probably eventually persuaded Newsom to veto?

I feel like the core thing that we've seen in DC is that the Overton window has shifted, almost entirely as a result of AI capabilities getting better, and now people are both more receptive to some of these arguments and more willing to acknowledge their sympathy.

[-]So8res17d*4936

To be clear, my recommendation for SB-1047 was not "be basically the same bill but talk about extinction risks and levy a few more restrictions on the labs", but rather "focus very explicitly on the extinction threat; say 'this bill is trying to address a looming danger described by a variety of scientists and industry leaders' or suchlike, shape the bill differently to actually address the extinction threat straightforwardly".

I don't have a strong take on whether SB-1047 would have been more likely to pass in that world. My recollection is that, back when I attempted to give this advice, I said I thought it would make the bill less likely to pass but more likely to have good effects on the conversation (in addition to it being much more likely to matter in cases where it did pass). But that could easily be hindsight bias; it's been a few years. And post facto, the modern question of what is "more likely" depends a bunch on things like how stochastic you think Newsom is (we already observed that he vetoed the real bill, so I think there's a decent argument that a bill with different content has a better chance even if it's a lower than our a-priori odds on SB-1047), though that's a... (read more)

5Buck16d

Ok. I agree with many particular points here, and there are others that I think are wrong, and others where I'm unsure. For what it's worth, I think SB-1047 would have been good for AI takeover risk on the merits, even though (as you note) it isn't close to all we'd want from AI regulation.

2Buck8d

Yeah to reiterate, idk why you think My guess is that the main reason they broadcast uncertainty is because they're worried that their position is unpalatable, rather than because of their internal sense of uncertainty.

[-]Richard_Ngo7d18-6

FWIW I broadcast the former rather than the latter because from the 25% perspective there are many possible worlds which the "stop" coalition ends up making much worse, and therefore I can't honestly broadcast "this is ridiculous and should stop" without being more specific about what I'd want from the stop coalition.

A (loose) analogy: leftists in Iran who confidently argued "the Shah's regime is ridiculous and should stop". It turned out that there was so much variance in how it stopped that this argument wasn't actually a good one to confidently broadcast, despite in some sense being correct.

[-]peterbarnett7d3613

Maybe it’s hard to communicate nuance, but it seems like there's a crazy thing going on where many people in the AI x-risk community think something like “Well obviously I wish it would stop, and the current situation does seem crazy and unacceptable by any normal standards of risk management. But there’s a lot of nuance in what I actually think we should do, and I don’t want to advocate for a harmful stop.”

And these people end up communicating to external people something like “Stopping is a naive strategy, and continuing (maybe with some safeguards etc) is my preferred strategy for now.”

This seems to miss out the really important part where they would actually want to stop if we could, but it seems hard and difficult/nuanced to get right.

[-]Richard_Ngo7d152

Yeah, I agree that it's easy to err in that direction, and I've sometimes done so. Going forward I'm trying to more consistently say the "obviously I wish people just wouldn't do this" part.

Though note that even claims like "unacceptable by any normal standards of risk management" feel off to me. We're talking about the future of humanity, there is no normal standard of risk management. This should feel as silly as the US or UK invoking "normal standards of risk management" in debates over whether to join WW2.

0Ronny Fernandez8d

To check, do you have particular people in mind for this hypothesis? Seems kinda rude to name them here, but could you maybe send me some guesses privately? I currently don't find this hypothesis as stated very plausible, or like sure maybe, but I think it's a relatively small fraction of the effect.

[-]Knight Lee18d103

Sorry for budging in, but I can't help but notice I agree with both what you and So8res are saying, but I think you aren't arguing about the same thing.

You seem to be talking about the dimension of "confidence," "obviousness," etc. and arguing that most proponents of AI concern seem to have enough of it, and shouldn't increase it too much.

So8res seems to be talking about another dimension which is harder to name. "Frank futuristicness" maybe? Though not really.

If you adjust your "frank futuristicness" to an absurdly high setting, you'll sound a bit crazy. You'll tell lawmakers "I'm not confident, and this isn't obvious, but I think that unless we pause AI right now, we risk a 50% chance of building a misaligned superintelligence. It might use nanobots to convert all the matter in the universe into paperclips, and the stars and galaxies will fade one by one."

But if you adjust your "frank futuristicness" to an absurdly low setting, you'll end up being ineffective. You'll say "I am confident, and this is obvious: we should regulate AI companies more because they are less regulated than other companies. For example the companies which research vaccines have to jump through so many clin... (read more)

5Buck17d

Ok; what do you think of Soares's claim that SB-1047 should have been made stronger and the connections to existential risk should have been made more clearly? That seems probably false to me.

[-]Richard_Ngo16d117

I think that if it were to go ahead, it should have been made stronger and clearer. But this wouldn't have been politically feasible, and therefore if that were the standard being aimed for it wouldn't have gone ahead.

This I think would have been better than the outcome that actually happened.

5Knight Lee17d

To be honest, I don't know. All I know is that a lot of organizations seem to be shy talking about the AI takeover risk, and the endorsements the book got surprised me, regarding how receptive government officials are. (Considering how little cherry-picking they did.) My very uneducated guess is that Newsom vetoed the bill because he was more of a consequentialist/Longtermist than the cause-driven lawmakers who passed the bill, so one can argue the failure mode was a "lack of appeal to consequentialist interests." One might argue "it passed by cause-driven lawmakers by a wide margin, but got blocked by the consequentialist." But the cause-driven vs. consequentialist motives are pure speculation, I know nothing about these people aside from Newsom's explanation...

[-]Rob Bensinger17d*1812

(Considering how little cherry-picking they did.)

From my perspective, FWIW, the endorsements we got would have been surprising even if they had been maximally cherry-picked. You usually just can't find cherries like those.

[-]Ae_rys18d363

Just commenting to say this post helped me build the courage to talk about AI risks around me more confidently and to stop thinking I needed to understand everything about it before saying I am deeply worried. Thanks Soares!

[-]Outsideobsserver18d295

Hi there,

I just made an account to offer my anecdotes as an outside observer. I admit, I’m not well versed in the of AI, I’m only a college graduate with a focus on biomedicine. But as a person who found his way to this site following the 2023 letter about AI extinction, I’ve nearly constantly read as much information as I could process to understand the state and stakes of the AI Risk conversation. For what it’s worth, I applaud those who have the courage to speak about something that in the grand scheme of things is hard to discuss with people without sounding condescending, apathetic or crazy. For someone like me, who has trouble with expressing the blood chilling dread I feel reading AI related headlines, I have nothing but the upmost respect for those who may have an influence in guiding this timeline in to one that succeeds.

I just feel like the communication form well informed people gets marred buy the sense of incapabilty. What I mean by that is the average layperson will engage with a meme or a light post about the risks of AI, but then swiftly become disillusioned by the prosepect of not being able to influence anything. (An aside, this is essentially the same... (read more)

[-]Haiku17d169

It's important for everyone to know that there are many things they can personally do to help steer the world onto a safer trajectory. I am a volunteer with PauseAI, and while I am very bad at grassroots organizing and lobbying, I still managed to start a local chapter and have some positive impact on my federal representative's office.

I suggest checking out PauseAI's Take Action page. I also strongly endorse ControlAI, who are more centralized and have an excellent newsletter.

6Outsideobsserver16d

I appreciate your comment, and the work that the individuals of both organizations are attempting to do now, but I can’t help but feel my point was slightly missed. It’s not that I wouldn’t support these organizations, or even the majority of laypersons would, it’s more simply that there’s a fatigue that comes with the style of protest and outreach these organizations do. There’s only so many dire warnings, so many pickett lines, so many cold calls and outreach before your target audiences becomes either overwhelmed and spirals in to a bout of severe depressions and anxiety (as I have many of times), or over saturated and dismissive since they see it as yet another failed dream of the naive. Unfortunately, years upon years of protest and subsequent media portrayal of protests have supplied people with a bitter pill of indifference and massaged the thoughts of people who could do things into cynical skeptics. If one wants the Overton window to swing to in a way that could archive meaningful change, the masses must have both hope and direction in which to change things while also striking a delicate balance of being grounded while anticipating greater risks. If you lead with “hey there’s a reasonable chance that this hyper intelligent software will have the ability to completely dismantle the planet into atoms in a microscopic timescale, so we need to stop it now and then point humanity into manifest destiny into the cosmos as the rightful stewards of the stars” people will blankly stare at you with bewilderment. But if you instead start by addressing things like job risks, deepfakes, concentration of power and the totalitarianism, tangible real issues people can see now, they may begin to open that door and then be more susceptible to discussing and acting on existential risk because they have the momentum behind them. Any movement, grassroots or grand, needs to have the momentum to slog through the doubts and denial, and I feel like if just a few minor sparks

7J Bostock12d

I spent approximately a year at PauseAI (Global) soul-searching over this, and I've come to the conclusion that this strategy is not a good idea. This is something I've changed my mind about. My original view was something like: "If we convince people to get on board with pausing AI for any reason, they'll eventually come round to the extinction risk concerns, in the same way that people who become vegan for e.g. environmental reason usually come around to the moral concerns. This is more efficient than trying to hit people with the extinction risk concerns from the start, since more people will be open to listening to non-extinction concerns." I think this is wrong. Recruiting people for non-extinction reasons was harder than I expected. I remember at one point I found a facebook group with 80K people called "Artists Against Generative AI" and got the organizers to share our stuff there and we literally got zero uptake from that. We did a few media campaigns on copyright grounds, and we didn't get much attention on that. I'm still not sure why this was the case, but we just didn't make headway. We didn't get any wins to leverage, we didn't build any momentum. And even if we did, we would have been pointed in the wrong direction. I now think something like this: "Everything we do should have the threat of extinction front-and-centre. We might protest about specific things, but this is 'wave' content in the MIRI sense (I don't know if MIRI is still doing this, I haven't actually seen them put out any 'wave' content but I could easily have missed it) and needs to be fed back into extinction threat concerns. Everything we talk about that isn't about extinction is in some sense a compromise to The Current Moment, and we should be careful of this." Example: we recently protested about DeepMind breaking their commitments from the Seoul summit. Whether or not they keep their commitments is probably not an X-risk lynchpin, but it is something that's happening now, and

5Outsideobsserver7d

Hi there! I apologize for not responding to this very insightful comment, I really appreciate your perspective on my admittedly scatter brained thought parent comment. Your comment definitely has caused me to reflect a-bit on my own, and updated me away slightly from my original position. I feel I may have been a bit ignorant to the actual state of PauseAI, as like I said in my original comments and replies it felt like an organization dangerously close to becoming orphaned from people’s thought processes. I’m glad to hear there are some ways around the issue I described. Maybe write a top level post about how this shift in understanding is benefiting your messaging to the general public? It may inform others of novel ways to spread a positive movement.

[-]Three-Monkey Mind19d25-5

Politicians were not understanding the bill as being about extinction threats; they were understanding the bill as being about regulatory capture of a normal budding technology industry.

Whether the bill is about extinction threats is nearly irrelevant; what’s important is what its first- and second-order effects will be. Any sensible politician will know of all kinds of cases where a bill’s (or constitutional amendment’s) full effects were either not known or explicitly disavowed by the bill’s proponents.

If you want to quell China fears here, you need to make the in-PRC push for this book a peer to the push for the American English one. Of course, the PRC takes a dim view of foreigners trying to bolster social movements, so — and I’m trying to not say this flippantly — good luck with that.

(I also think pushing this book’s pile of memes in China is a good idea because they’re probably about as capable of making AGI as we are, but that’s a separate issue.)

9J Bostock12d

How would you recommend pushing this book's pile of memes in China? My first thought would be trying to organize some conference with non-Chinese and (especially) Chinese experts, the kinds who advise the government, centered around the claims of the book. I don't know how the CCP would view this though, I'm not an expert on Chinese internal politics.

2Three-Monkey Mind11d

I have no idea whatsoever, sorry.

[-]jvican14d2216

AI risk discussions are happening at elite and EU institutional levels, but in my experience they're not reaching regular European citizens, especially in Spain (where I'm originally from).

This is a major problem because politicians respond to what their constituents care about, and if people aren't talking about it, they won't demand action on it.

The crux of the issue is that most Europeans simply aren't familiar with AI risk arguments, so they're not discussing or prioritizing these issues among their circles. Without this kind of public awareness, further political action from the EU is unlikely, and will likely wane as AI gets more and more important in the economy.

I'd like to encourage you to translate the book into Spanish, French, German, and Italian could help bridge this gap.

Spanish would be especially valuable given its global reach, and how under-discussed these risks are among Spanish society. But the same point extends to other EU countries. There's almost no awareness of AI risks among citizens, and this book could change that completely.

[-]Knight Lee18d227

I really like this.

I think AI concern can fail in two ways:

We lose the argument. We confidently state our confident positions, with no shyness or sugarcoating. It grabs a lot of attention, but it's negative attention and ridicule. We get a ton of engagement, by those who consider us a low status morbid curiosity. We make a lot of powerful enemies, who relentlessly attack AI concern and convince almost everyone.
We don't lose any argument, but the argument never happens. We have a ton of impressive endorsements, and anyone who actually reads about the drama learn that our side consists of high status scientists and geniuses. We have no enemies--the only people rarer than someone arguing for AI risk is someone arguing against AI risk. And yet... we are ignored. Politicians are simply too busy to think about this. They may think "I guess your logic is correct... but no other politicians seem to be invested in this, and I don't really want to be the first one."

Being bolder increases the "losing argument" risk but decreases the "argument never happens" risk. And this is exactly what we want at this point in time. (As long as you don't do negative actions like traffic obstruction protests... (read more)

[-]Damian Tatum14d179

It seems like I see this sort of thing (people being evasive or euphemistic when discussing AI risk) a LOT. Even discussions by the AI2027 authors, who specifically describe a blow by blow extinction scenario that ends with AI killing 99% of humans with a bioweapon and then cleaning up the survivors with drones, frequently refer to it with sanitized language like "it could move against humans" [1] or "end up taking over" [2]. Guys you literally wrote a month by month extinction scenario with receipts! No need to protect our feelings here!

https://open.substack.com/pub/astralcodexten/p/introducing-ai-2027?utm_source=share&utm_medium=android&r=qmni)
https://www.dwarkesh.com/p/scott-daniel

[-]boazbarak19d172

I haven’t read the book (though pre ordered it!) but I don’t think the statement in its title is true. I agree however with this post that people should be honest about their beliefs.

[-]FVelde19d175

I can attest that for me talking about AI dangers in an ashamed way has rarely if ever prompted a positive response. I've noticed and been told that it gives 'intellectual smartass' vibes rather than 'concerned person' vibes.

[-]geoffreymiller14d16-22

A plea for having the courage to morally stigmatize the people working in the AGI industry:

I agree with Nate Soares that we need to show much more courage in publicly sharing our technical judgments about AI risks -- based on our understanding of AI, the difficulties of AI alignment, the nature of corporate & geopolitical arms races, the challenges of new technology regulation & treaties, etc.

But we also need to show much more courage in publicly sharing our social and moral judgments about the evils of the real-life, flesh-and-blood people who are driving these AI risks -- specifically, the people leading the AI industry, working in it, funding it, lobbying for it, and defending it on social media.

Sharing our technical concerns about these abstract risks isn't enough. We also have to morally stigmatize the specific groups of people imposing these risks on all of us.

We need the moral courage to label other people evil when they're doing evil things.

If we don't do this, we look like hypocrites who don't really believe that AGI/ASI would be dangerous.

Moral psychology teaches us that moral judgments are typically attached not just to specific actions, or to emergen... (read more)

[-]TsviBT12d4228

Sharing our technical concerns about these abstract risks isn't enough. We also have to morally stigmatize

I'm with you up until here; this isn't just a technical debate, it's a moral and social and political conflict with high stakes, and good and bad actions.

the specific groups of people imposing these risks on all of us.

To be really nitpicky, I technically agree with this as stated: we should stigmatize groups as such, e.g. "the AGI capabilities research community" is evil.

We need the moral courage to label other people evil when they're doing evil things.

Oops, this is partially but importantly WRONG. From Braχot 10a:

With regard to the statement of Rabbi Yehuda, son of Rabbi Shimon ben Pazi, that David did not say Halleluya until he saw the downfall of the wicked, the Gemara relates: There were these hooligans in Rabbi Meir’s neighborhood who caused him a great deal of anguish. Rabbi Meir prayed for God to have mercy on them, that they should die. Rabbi Meir’s wife, Berurya, said to him: What is your thinking? On what basis do you pray for the death of these hooligans? Do you base yourself on the verse, as it is written: “Let sins cease from the land” (Psalms 104:35)

... (read more)

5geoffreymiller6d

TsviBT - thanks for a thoughtful comment. I understand your point about labelling industries, actions, and goals as evil, but being cautious about labelling individuals as evil. But I don't think it's compelling. You wrote 'You're closing off lines of communication and gradual change. You're polarizing things.' Yes, I am. We've had open lines of communication between AI devs and AI safety experts for a decade. We've had pleas for gradual change. Mutual respect, and all that. Trying to use normal channels of moral persuasion. Well-intentioned EAs going to work inside the AI companies to try to nudge them in safer directions. None of that has worked. AI capabilities development is outstripping AI safety developments at an ever-increasing rate. The financial temptations to stay working inside AI companies keep increasing, even as the X risks keep increasing. Timelines are getting shorter. The right time to 'polarize things' is when we still have some moral and social leverage to stop reckless ASI development. The wrong time is after it's too late. Altman, Amodei, Hassabis, and Wang are buying people's souls -- paying them hundreds of thousands or millions of dollars a year to work on ASI development, despite most of their workers they supervise knowing that they're likely to be increasing extinction risk. This isn't just a case of 'collective evil' being done by otherwise good people. This is a case of paying people so much that they ignore their ethical qualms about what they're doing. That makes the evil very individual, and very specific. And I think that's worth pointing out.

[-]Ben Pace14d229

(This rhetoric is not quite my rhetoric, but I want to affirm that I do believe that ~most people working at big AI companies are contributing to the worst atrocity in human history, are doing things that are deontologically prohibited, and are morally responsible for that.)

7geoffreymiller13d

Ben -- so, we're saying the same things, but you're using gentler euphemisms. I say 'evil'; you say 'deontologically prohibited'. Given the urgency of communicating ASI extinction risks to the public, why is this the time for gentle euphemisms?

[-]Ben Pace13d1812

For one, I think I'm a bit scared of regretting my choices. Like, calling someone evil and then being wrong about it isn't something where you just get to say "oops, I made a mistake" afterwards, you did meaningfully move to socially ostracize someone, mark them as deeply untrustworthy, and say that good people should remove their power, and you kind of owe them something significant if you get that wrong.

For two, a person who has done evil, versus a person who is evil, are quite different things. I think that it's sadly not always the case that a person's character is aligned with a particular behavior of theirs. I think it's not accurate to think of all the people building the doomsday machines as generically evil and someone who will do awful things in lots of different contexts, I think there's a lot of variation in the people and their psychologies and predispositions, and some are screwing up here (almost unforgivably, to be clear) in ways they wouldn't screw up in different situations.

[-]Adam Scholl13d139

For two, a person who has done evil, versus a person who is evil, are quite different things. I think that it's sadly not always the case that a person's character is aligned with a particular behavior of theirs.

I do think many of the historical people most widely considered to be evil now were similarly not awful in full generality, or even across most contexts. For example, Eichmann, the ops lead for the Holocaust, was apparently a good husband and father, and generally took care not to violate local norms in his life or work. Yet personally I feel quite comfortable describing him as evil, despite "evil" being a fuzzy folk term of the sort which tends to imperfectly/lossily describe any given referent.

[-]Ben Pace13d*141

I'm not quite sure what I make of this, I'll take this opportunity to think aloud about it.

I often take a perspective where most people are born a kludgey mess, and then if they work hard they can become something principled and consistent and well-defined. But without that, they don't have much in the way of persistent beliefs or morals such that they can be called 'good' or 'evil'.

I think of an evil person as someone more like Voldemort in HPMOR, who has reflected on his principles and will be persistently a murdering sociopath, than someone who ended up making horrendous decisions but wouldn't in a different time and place. I think if you put me under a lot of unexpected political forces and forced me to make high-stakes decisions, I could make bad decisions, but not because I'm a fundamentally bad person.

I do think it makes sense to write people off as bad people, in our civilization. There are people who have poor impulse control, who have poor empathy, who are pathological liars, and who aren't save-able by any of our current means, and will always end up in jail or hurting people around them. I rarely interact with such people so it's hard for me to keep this in ... (read more)

5Said Achmiz12d

Eichmann was definitely evil. The popular conception of Eichmann as merely an ordinary guy who was “just doing his job” and was “boring” is partly mischaracterization of Arendt’s work, partly her own mistakes (i.e., her characterizations, which are no longer considered accurate by historians). An example of the former: (Hmm… that last line is rather reminiscent of something, no?) Concerning the latter:

6boazbarak12d

I am one of those people that are supposed to be stigmatized/detterred by this action. I doubt this tactic will be effective. This thread (including the disgusting comparison to Eichmann who directed the killing of millions in the real world - not in some hypothetical future one) does not motivate me to interact with the people holding such positions. Given that much of my extended family was wiped out by the holocaust, I find these Nazi comparisons abhorrent, and would not look forward to interact with people making them whether or not they decide to boycott me. BTW this is not some original tactic, PETA is using similar approaches for veganism. I don’t think they are very effective either. To @So8res - I am surprised and disappointed that this Godwin’s law thread survived a moderation policy that is described as “Reign of Terror”

[-]habryka12d4030

I've often appreciated your contributions here, but given the stakes of existential risk, I do think that if my beliefs about risk from AI are even remotely correct, then it's hard to escape the conclusion that the people presently working at labs are committing the greatest atrocity that anyone in human history has or will ever commit.

The logic of this does not seem that complicated, and while I disagree with Geoffrey Miller on how he goes about doing things, I have even less sympathy for someone reacting to a bunch of people really thinking extremely seriously and carefully about whether what that person is doing might be extremely bad with "if people making such comparisons decide to ostracize me then I consider it a nice bonus". You don't have to agree, but man, I feel like you clearly have the logical pieces to understand why one could believe you are causing extremely great harm, without that implying the insanity of the person believing that.

I respect at least some of the people working at capability labs. One thing that unites all of the ones I do respect is that they treat their role at those labs with the understanding that they are in a position of momentous responsibility, and that them making mistakes could indeed cause historically unprecedented levels of harm. I wish you did the same here.

[-]boazbarak12d113

I edited the original post to make the same point with less sarcasm.

I take risk from AI very seriously which is precisely why I am working in alignment at OpenAI. I am also open to talking with people having different opinions, which is why I try to follow this forum (and also preordered the book). But I do draw the line at people making Nazi comparisons.

FWIW I think radicals often hurt the causes they espouse, whether it is animal rights, climate change, or Palestine. Even if after decades the radicals are perceived to have been on “the right side of history”, their impact was often negative and it caused that to have taken longer: David Shor was famously cancelled for making this point in the context of the civil rights movement.

9Ben Pace11d

Sorry to hear the conversation was on a difficult topic for you; I imagine that is true for many of the Jewish folks we have around these parts. FWIW I think we were discussing Eichmann in order to analyze what 'evil' is or isn't, and did not make any direct comparisons between him and anyone. ...oh, now I see what Said's "Hmm… that last line is rather reminiscent of something, no?" is probably making such a comparison (I couldn't tell what he meant it when I read it initially). I can see why you'd respond negatively to that. While there's a valid point to be made about how people who just try to gain status/power/career-capital without thinking about ethics can do horrendous things, I do not think that it is healthy for discourse to express that in the passive-aggressive way that Said did.

[-]Said Achmiz11d2839

The comparisons invite themselves, frankly. “Careerism without moral evaluation of the consequences of one’s work” is a perfect encapsulation of the attitudes of many of the people who work in frontier AI labs, and I decline to pretend otherwise.

(And I must also say that I find the “Jewish people must not be compared to Nazis” stance to be rather absurd, especially in this sort of case. I’m Jewish myself, and I think that refusing to learn, from that particular historical example, any lessons whatsoever that could possibly ever apply to our own behavior, is morally irresponsible in the extreme.)

EDIT: Although the primary motivation of my comment about Eichmann was indeed to correct the perception of the historians’ consensus, so if you prefer, I can remove the comparison to a separate comment; the rest of the comment stands without that part.

8Ben Pace11d

I agree with your middle paragraph. To be clear, I would approve more of a comment that made the comparison overtly[0], rather than one that made it in a subtle way that was harder to notice or that people missed (I did not realize what you were referring to until I tried to puzzle at why boaz had gotten so upset!). I think it is not healthy for people to only realize later that they were compared to Nazis. And I think it fair for them to consider that an underhanded way to cause them social punishment, to do it in a way that was hard to directly respond to. I believe it’s healthier for attacks[1] to be open and clear. [0] To be clear, there may still be good reasons to not throw in such a jab at this point in the conversation, but my main point is that doing it with subtlety makes it worse, not better, because it also feels sneaky. [1] “Attacks”, a word which here means “statements that declare someone has a deeply rotten character or whose behavior has violated an important norm, in a way that if widely believed will cause people to punish them”.

4Ben Pace11d

(I don’t mean to derail this thread with discussion of discussion norms. Perhaps if we build that “move discourse elsewhere button” that can later be applied back to this thread.)

5boazbarak11d

Thank you Ben. I don’t think name calling and comparisons are helpful to a constructive debate, which I am happy to have. Happy 4th!

1geoffreymiller6d

boazbarak -- I don't understand your implication that my position is 'radical'. I have exactly the same view on the magnitude of ASI extinction risk that every leader of a major AI company does -- that it's a significant risk. The main difference between them and me is that they are willing to push ahead with ASI development despite the significant risk of human extinction, and I think they are utterly evil for doing so, because they're endangering all of our kids. In my view, risking extinction for some vague promise of an ASI utopia is the radical position. Protecting us from extinction is a mainstream, commonsense, utterly normal human position.

[-]So8res12d192

(From a moderation perspective:

I consider the following question-cluster to be squarely topical: "Suppose one believes it is evil to advance AI capabilities towards superintelligence, on the grounds that such a superintelligence would quite likely to kill us all. Suppose further that one fails to unapologetically name this perceived evil as 'evil', e.g. out of a sense of social discomfort. Is that a failure of courage, in the sense of this post?"
I consider the following question-cluster to be a tangent: "Suppose person X is contributing to a project that I believe will, in the future, cause great harms. Does person X count as 'evil'? Even if X agrees with me about which outcomes are good and disagrees about the consequences of the project? Even if the harms of the project have not yet occurred? Even if X would not be robustly harmful in other circumstances? What if X thinks they're trying to nudge the project in a less-bad direction?"
I consider the following sort of question to be sliding into the controversy attractor: "Are people working at AI companies evil?"

The LW mods told me they're considering implementing a tool to move discussions to the open thread (so that they may continue without derailing the topical discussions). FYI @habryka: if it existed, I might use it on the tangents, idk. I encourage people to pump against the controversy attractor.)

[-]boazbarak12d102

I agree with you on the categorization of 1 and 2. I think there is a reason why Godwin’s law was created once thread follow the controversy attractor to this direction they tend to be unproductive.

3Knight Lee11d

I completely agree this discussion should be moved outside your post. But the counterintuitive mechanics of LessWrong mean a derailing discussion may actually increase the visibility and upvotes of your original message (by bumping it in the "recent discussion"). (It's probably still bad if it's high up in the comment section.) It's too bad you can only delete comment threads, you can't move them to the bottom or make them collapsed by default.

[-]TsviBT12d1818

The apparent aim of OpenAI (making AGI, even though we don't know how to do so without killing everyone) is evil.

8niplav6d

I agree that a comparison to Eichmann is not optimal. Instead, if AI turns out to have consequences so bad that they outweigh the good, it'll have been better to compare people working at AI labs to Thomas Midgley, who insisted that his leaded gasoline couldn't be dangerous even when presented with counter-evidence, and Edward Teller, who (as far as I can tell) was simply fascinated by the engineering challenges of scaling hydrogen bombs to levels that could incinerate entire continents. These two people still embody two archetypes of what could reasonably be called "evil", but arguably fit better with the psychology of people currently working at AI labs. These two examples also avoid Godwin's law type attractors.

3[comment deleted]12d

4Ben Pace11d

That's interesting to hear that many historians believe he was secretly more ideologically motivated than Arendt thought, and also believe that he portrayed a false face during all of the trials, thanks for the info.

4geoffreymiller13d

Ben -- your subtext here seems to be that only lower-class violent criminals are truly 'evil', whereas very few middle/upper-class white-collar people are truly evil (with a few notable exceptions such as SBF or Voldemort) -- with the implications that the majority of ASI devs can't possibly be evil in the ways I've argued. I think that doesn't fit the psychological and criminological research on the substantial overlap between psychopathy and sociopathy, and between violent and non-violent crime. It also doesn't fit the standard EA point that a lot of 'non-evil' people can get swept up in doing evil collective acts as parts of collectively evil industries, such as slave-trading, factory farming, Big Tobacco, the private prison system, etc. - but that often, the best way to fight such industries is to use moral stigmatization.

[-]Ben Pace13d*113

You mis-read me on the first point; I said that (something kind of like) 'lower-class violent criminals' are sometimes dysfunctional and bad people, but I was distinguishing that from someone more hyper competent and self-aware like SBF or Voldemort; I said that only the latter are evil. (For instance, they've hurt orders of magnitude more people.)

(I'm genuinely not sure what research you're referring to – I am expect you are 100x as familiar with the literature as I am, and FWIW I'd be happy to get a pointer or two of things to read.^[1])

The standard EA point is to use moral stigmatization? Even if that's accurate, I'm afraid I no longer have any trust in EAs to do ethics well. As an example that you will be sympathetic to, lots of them have endorsed working at AI companies over the past decade (but many many other examples have persuaded me of this point).

To be clear, I am supportive of moral stigma being associated with working at AI companies. I've shown up to multiple protests outside the companies (and I brought my mum!). If you have any particular actions in mind to encourage me to do (I'm probably not doing as much as I could) I'm interested to hear them. Perhaps you could w... (read more)

7geoffreymiller13d

Yes, it takes courage to call people out as evil, because you might be wrong, you might unjustly ruin their lives, you might have mistakenly turned them into scapegoat, etc. Moral stigmatization carries these risks. Always has. And people understand this. Which is why, if we're not willing to call the AGI industry leaders and devs evil, then people will see us failing to have the courage of our convictions. They will rightly see that we're not actually confident enough in our judgments about AI X-risk to take the bold step of pointing fingers and saying 'WRONG!'. So, we can hedge our social bets, and try to play nice with the AGI industry, and worry about making such mistakes. Or, we can save humanity.

[-]Ben Pace13d178

To be clear, I think it would probably be reasonable for some external body like the UN to attempt to prosecute & imprison ~everyone working at big AI companies for their role in racing to build doomsday machines. (Most people in prison are not evil.) I'm a bit unsure if it makes sense to do things like this retroactively rather than to just outlaw it going forward, but I think it sometime makes sense to prosecute atrocities after the fact even if there wasn't a law against it at the time. For instance my understanding is that the Nuremberg trials set precedents for prosecuting people for war crimes, crimes against humanity, and crimes against peace, even though legally they weren't crimes at the time that they happened.

I just have genuine uncertainty about the character of many of the people in the big AI companies and I don't believe they're all fundamentally rotten people! And I think language is something that can easily get bent out of shape when the stakes are high, and I don't want to lose my ability to speak and be understood. Consequently I find I care about not falsely calling people's character/nature evil when what I think is happening is that they are committing an atrocity, which is similar but distinct.

[-]Raemon13d1214

My answer to this is "because framing things in terms of evil turns the situation more mindkilly, not really the right gears, and I think this domain needs clarity-of-thought more than it needs a social-conflict orientation"

(I'm not that confident about that, and don't super object to other people calling them evil. But I think "they are most likely committing a great atrocity" is pretty non-euphemistic and more true)

4Ben Pace13d

Openly calling people evil has some element of "deciding who to be in social conflict with", but insofar as it has some element of "this is simply an accurate description of the world" then FWIW I want to note that this consideration partially cuts in favor of just plainly stating what counts as evil, even whether specific people have met that bar.

7Raemon13d

If I thought evil was a more useful gear here I'd be more into it. Like, I think "they probably committing an atrocity" carves reality at the joints. I think there are maybe 3 people involved with AGI development who seem like they might be best described as "evil" (one of whom is Sam Altman, who I feel comfortable naming because I think we've seen evidence of him doing nearterm more mundane evil, rather than making guesses about their character and motivations) I think it probably isn't helpful to think of Eichmann as evil, though again fine to say "he commited atrocities" or even "he did evil."

[-]Ben Pace13d114

Yeah that makes sense.

As an aside, I notice that I currently feel much more reticent to name individuals who there is not some sort of legal/civilizational consensus about their character. I think I am a bit worried about contributing to the dehumanization of people who are active players, and a drop in basic standards of decency toward them, even if I were to believe they were evil.

[-]Richard_Kennaway11d1611

We have to denounce them as the Bad Guys. As traitors to our species. And then, later, once they've experienced the most intense moral shame they've ever felt, [etc. contd. p.94]

This is self-indulgent, impotent fantasy. Everyone agrees that people hurting children is bad. People are split on whether AGI/ASI is an existential threat.^[1] There is no "we" beyond "people who agree with you". "They" are not going to have anything like the reaction you're imagining. Your strategy of screaming and screaming and screaming and screaming and screaming and screaming and screaming and screaming is not an effective way of changing anyone's mind.

Anyone responding "but it IS an existential threat!!" is missing the point. ↩︎

1geoffreymiller6d

Richard -- I think you're just factually wrong that 'people are split on whether AGI/ASI is an existential threat'. Thousands of people signed the 2023 CAIS statement on AI risk, including almost every leading AI scientist, AI company CEO, AI researcher, AI safety expert, etc. There are a few exceptions, such as Yann LeCun. And there are a few AI CEOs, such as Sam Altman, who had previously acknowledged the existential risks, but now downplay them. But if all the leading figures in the industry -- including Altman, Amodei, Hassabis, etc -- have publicly and repeatedly acknowledged the existential risks, why would you claim 'people are split'?

7Richard_Kennaway6d

You just mentioned LeCun and "a few AI CEOs, such as Sam Altman" as exceptions, so it isn't by any means "all the leading figures". I would also name Mark Zuckerberg, who has started "Superintelligence Labs" with the aim of "personal superintelligence for everyone", with nary a mention of how if anyone builds it, everyone dies. Presumably all the talent he's bought are on board with that. I also see various figures (no names to hand) pooh-poohing the very idea of ASI at all, or of ASI as existential threat. They may be driven by the bias of dismissing the possibility of anything so disastrous as to make them have to Do Something and miss lunch, but right or wrong, that's what they say. And however many there are on each side, I stand by my judgement of the futility of screaming shame at the other side, and of the self-gratifying fantasy about how "they" will react.

[-]Knight Lee13d*165

I'm personally against this as matter of principle, and I also don't think it'll work.

Moral stigmatizing only works against a captive audience. It doesn't work against people who can very easily ignore you.
- You're more likely to stop eating meat if a kind understanding vegetarian/vegan talks to you and makes you connect with her story of how she stopped eating meat. You're more likely to simply ignore a militant one who calls you a murderer.
Moral stigmatizing failed to stop nuclear weapon developers, even though many of them were the same kind of "nerd" as AI researchers.
- People see Robert Oppenheimer saying "Now, I am become Death, the destroyer of worlds" as some morally deep stuff. "The scientific community ostracized [Edward] Teller," not because he was very eager to build bigger bombs (like the hydrogen bomb and his proposed Sundial), but because he made Oppenheimer lose his security clearance by saying bad stuff about him.
Which game do you choose to play? The game of dispassionate discussion, where the truth is on your side? Or the game of Twitter-like motivated reasoning, where your side looks much more low status than the AI lab people, and the status quo is certainly not on

... (read more)

[-]geoffreymiller13d115

Hi Knight, thanks for the thoughtful reply.

I'm curious whether you read the longer piece about moral stigmatization that I linked to at EA Forum? It's here, and it addresses several of your points.

I have a much more positive view about the effectiveness of moral stigmatization, which I think has been at the heart of almost every successful moral progress movement in history. The anti-slavery movement stigmatized slavery. The anti-vivisection movement stigmatized torturing animals for 'experiments'. The women's rights movement stigmatized misogyny. The gay rights movement stigmatized homophobia.

After the world wars, biological and chemical weapons were not just regulated, but morally stigmatized. The anti-landmine campaign stigmatized landmines.

Even in the case of nuclear weapons, the anti-nukes peace movement stigmatized the use and spread of nukes, and was important in nuclear non-proliferation, and IMHO played a role in the heroic individual decisions by Arkhipov and others not to use nukes when they could have.

Regulation and treaties aimed to reduce the development, spread, and use of Bad Thing X, without moral stigmatization of Bad Thing X, doesn't usually work very... (read more)

3Knight Lee11d

That's a very good point, and these examples really changes my intuition from "I can't see this being a good idea," to "this might make sense, this might not, it's complicated." And my earlier disagreement mostly came from my intuition. I still have disagreements, but just to clarify I now agree your idea deserves more attention that it's getting. My remaining disagreement is I think stigmatization only reaches the extreme level of "these people are literally evil and vile," after the majority of people already agree. In places in India where the majority of people are already vegetarians, and already feel that eating meat is wrong, the social punishment of meat eaters does seem to deter them. But in places where most people don't think eating meat is wrong, prematurely calling meat eaters evil may backfire. This is because you've created a "moral-duel" where you force outside observers to either think the meat-eater is the bad guy, or you're the bad guy (or stupid guy). This "moral-duel" drains the moral standing of both sides. If you're near the endgame, and 90% of people already are vegetarians, then this moral-duel will first deplete the meat-eater's moral standing, and may solidify vegetarianism. But if you're at the beginning, when only 1% of people support your movement. You desperately want to invest your support and credibility into further growing your support and credibility, rather than burning it in a moral-duel against the meat-eater majority the way militant vegans did. Nurturing credibility is especially important for AI Notkilleveryoneism, where the main obstacle is a lack of credibility and "this sounds like science fiction." Finally, at least only go after the AI lab CEOs, as they have relatively less moral standing, compared to the rank and file researchers. E.g. in this quicktake Mikhail Samin appealed to researchers as friends asking them to stop "deferring" to their CEO. Even for nuclear weapons, biological weapons, chemical weapons,

2geoffreymiller6d

Knight -- thanks again for the constructive engagement. I take your point that if a group is a tiny and obscure minority, and they're calling the majority view 'evil', and trying to stigmatize their behavior, that can backfire. However, the surveys and polls I've seen indicate that the majority of humans already have serious concerns about AI risks, and in some sense are already onboard with 'AI Notkilleveryoneism'. Many people are under-informed or misinformed in various ways about AI, but convincing the majority of humanity that the AI industry is acting recklessly seems like it's already pretty close to feasible -- if not already accomplished. I think the real problem here is raising public awareness about how many people are already on team 'AI Notkilleveryoneism' rather than team 'AI accelerationist'. This is a 'common knowledge' problem from game theory -- the majority needs to know that they're in the majority, in order to do successful moral stigmatization of the minority (in this case, the AI developers).

1Knight Lee5d

Haha you're right, in another comment I was saying To be honest, I'm extremely confused. Somehow, AI Notkilleveryoneism... is both a tiny minority and a majority at the same time. That makes sense, it seems to explain things. The median AI expert also has a 5% to 10% chance of extinction, which is huge. I'm still not in favour of stigmatizing AI developers, especially right now. Whether AI Notkilleveryoneism is a real minority or an imagined minority, if it gets into a moral-duel with AI developers, it will lose status, and it will be harder for it to grow (by convincing people to agree with it, or by convincing people who privately agree to come out of the closet). People tend to follow "the experts" instead of their very uncertain intuitions about whether something is dangerous. With global warming, the experts were climatologists. With cigarette toxicity, the experts were doctors. But with AI risk, you were saying that, It sounds like, the expertise people look to when deciding "whether AI risk is serious or sci-fi," comes from leading AI scientists, and even AI company CEOs. Very unfortunately, we may depend on our good relations with them... :(

9Tom Bibby12d

Moral ostracisation of factory farmers is somewhat ineffective because the vast majority of people are implicated in factory farming. They fund it every day and view eating animals as a key part of their identity. Calling factory farming murder/torture is calling nearly every member of the public a murderer/torturer. (Which may be true but is unlikely to get them to change their habits) Calling the race to ASI murder is only calling AI researchers and funders murderers. The general public are not morally implicated and don't view use of AI as a key part of their identity. The polling shows that they're not on board with the pace of AI development, think it poses a significant risk of human extinction, and that they don't trust the CEOs of AI companies to act responsibly.

4Knight Lee9d

That's a very good point, and I didn't really analyze the comparison. I guess maybe meat eating isn't the best comparison. The closest comparison might be researchers developing some other technology, which maybe 2/3 people see as a net negative. E.g. nuclear weapons, autonomous weapons, methods for extracting fossil fuel, tobacco, etc. But no campaign even really tried to stigmatize these researchers. Every single campaign against these technologies have targeted the companies, CEOs, or politicians leading them, without really any attack towards the researchers. Attacking them is sort of untested.

[-]Drake Thomas8d12-6

I think this is a pretty unhelpful frame. Most people working at an AI lab are somewhere between "person of unremarkable moral character who tells themselves a vague story about how they're doing good things" and "deeply principled person trying their best to improve the world as best they can". I think working at an AI lab requires less failure of moral character than, say, working at a tobacco company, for all that the former can have much worse effects on the world.

There are a few people I think it is fair to describe as actively morally bad, and willfully violating deontology - it seems likely to me that this is true of Sam Altman, for instance - but I think "evil" is just not a very helpful word here, will not usefully model the actions of AI lab employees, and will come across as obviously disingenuous to anyone who hears such rhetoric if they actually interact with any of the people you're denigrating. If you had to be evil to end the world, the world would be a lot safer!

I think it's fine and good to concentrate moral opprobrium at specific actions people take that are unprincipled or clear violations of deontology - companies going back on commitments, people taking on rol... (read more)

[-]Ben Pace8d156

But the mere choice of job title is usually not a deontology violation for these people, because they don't think it has the harms to the world you think it does!

I don't think this step is locally valid? Or at least, in many situations, I don't think ignorance of the consequences if your actions absolves you of responsibility for them.

As an example, if you work hard to help elect a politician who you believe was principled and good, and then when they get into office they're a craven sellout who causes thousands of people to die, you bear some responsibility for it and for cleaning up your mess. As another example, if you work hard at a company and then it turns out the company is a scam and you've stolen money from all your customers, you bear some responsibility to clean up the mess and help the people whose lives your work ruined.

Relatedly, it is often the case that the right point to apply liability is when someone takes an action with a lot of downside, regardless of intent. Here are some legal examples a shoggoth gave me of holding people accountable even if they didn't know the harm they were causing.

A company can be liable for harm caused by its products even if it followed

... (read more)

5Drake Thomas8d

Yeah, sorry - I agree that was a bit sloppy of me. I think it is very reasonable to accuse people working at major AI labs of something like negligence / willful ignorance, and I agree that can be a pretty serious moral failing (indeed I think it's plausibly the primary moral failing of many AI lab employees). My objection is more to the way the parent comment is connoting "evil" just from one's employer leading to bad outcomes as if those outcomes are the known intent of such employees.

[-]geoffreymiller6d114

Drake -- this seems like special pleading from an AI industry insider.

You wrote 'I think working at an AI lab requires less failure of moral character than, say, working at a tobacco company, for all that the former can have much worse effects on the world.'

That doesn't make sense to me. Tobacco kills about 8 million people a year globally. ASI could kill about 8 billion. The main reason that AI lab workers think that their moral character is better than that of tobacco industry workers is that the tobacco industry has already been morally stigmatized over the last several decades -- whereas the AI industry has not yet been morally stigmatized in proportion to its likely harms.

Of course, ordinary workers in any harm-imposing industry can always make the argument that they're good (or at least ethically mediocre) people, that they're just following orders, trying to feed their families, weren't aware of the harms, etc.

But that argument does not apply to smart people working in the AI industry -- who have mostly already been exposed to the many arguments that AGI/ASI is a uniquely dangerous technology. And their own CEOs have already acknowledged these risks. And yet people continue to work in this industry.

Maybe a few workers at a few AI companies might be having a net positive impact in reducing AI X-risk. Maybe you're one of the lucky few. Maybe.

[-]habryka6d1910

ASI could kill about 8 billion.

The future is much much bigger than 8 billion people. Causing the extinction of humanity is much worse than killing 8 billion people. This really matters a lot for arriving at the right moral conclusions here.

3sjadler1d

Could you say a bit more about why you view the “extinction >>> 8B” as so important? I’d have assumed that at your P(extinction), even treating extinction as just 8B deaths still vastly outweighs the possible lives saved from AI medical progress?

[-]habryka13h170

I don't think it's remotely as obvious then! If you don't care about future people, then your key priority is to achieve immortality for the current generation, for which I do think building AGI is probably your best bet.

If it were to take 50+ years to build AGI, that would imply most people on earth have died of aging, and so you should have probably just rushed towards AGI if you think that would have been less than 50% likely to cause extinction.

People who hold this position are arguing for things like "we should only slow down AI development if for each year of slowing down we would be reducing risk of human extinction by more than 1%", which is a policy that if acted on consistently would more likely than not cause humanity's extinction within 100 years (as you would be accepting a minimum of a 1% chance of death each year in exchange for faster AI development).

Here are ChatGPTs actuarial tables about how long the current population is expected to survive:

Years into the future	Share of people alive today who are expected to die*	Rough number of deaths (out of 8.23 billion)
10 years (2035)	≈ 8 %	~0.65 billion
20 years (2045)	≈ 25 %	~2.0 billion
30 years (2055)	≈ 49 %	~4.0 billion
40

... (read more)

4sjadler12h

Thanks for explaining that, really appreciate it! One thing I notice I’d been assuming: that “8B-only” people would have a policy like “care about the 8B people who are living today, but also the people in say 20 years who’ve been born in the intervening time period.” But that’s basically just a policy of caring about future people! Because there’s not really a difference between “future people at the point that they’ve actually been born” and “future people generally”

5Drake Thomas6d

I agree with you that a typical instance of working at an AI lab has worse consequences in expectation than working at a tobacco company, and I think that for a person who shares all your epistemic beliefs to work in a typical role at an AI lab would indeed be a worse failure of moral character than to work at a tobacco company. I also agree that in many cases people at AI labs have been exposed at least once to arguments which, if they had better epistemics and dedicated more time to thinking about the consequences of their work, could have convinced them that it was bad for the world for them to do such work. And I do think the failure to engage with such arguments and seriously consider them, in situations like these, is a stain on someone's character! But I think it's the sort of ethical failure which a majority of humans will make by default, rather than something indicative of remarkably bad morality. I just don't think this sort of utilitarian calculus makes sense to apply when considering the actions of people who don't share the object-level beliefs at hand! I think people who worked to promulgate communism in the late 19th century were not unusually evil, for instance.

2Garrett Baker6d

This also seems locally invalid. Most people in fact don’t make this ethical failure because they don’t work at AI labs, nor do they dedicate their lives to work which has nearly as much power or influence on others as this. It does seem consistent (and agree with commonsense morality) to say that if you are smart enough to locate the levers of power in the world, and you pursue them, then you have a moral responsibility to make sure you use them right if you do get your hands on them, otherwise we will call you evil and grossly irresponsible.

-5Shankar Sivarajan12d

[-]Ronny Fernandez8d90

Curated. I have been at times more cautious in communicating my object level views than I now wish I had been. I appreciate this post as a flag for courage: something others might see, and which might counter some of the (according to me) prevailing messages of caution. Those messages, at least in my case, significantly contributed to my caution, and I wish there had been something like this post around for me to read before I had to decide how cautious to be.

The argument this post presents for the conclusion that many people should be braver in communicat... (read more)

[-]jamiefisher7d80

"We cold-emailed a bunch of famous people..."

"Matt Stone", co-creator of South Park? Have you tried him?

He's demonstrated interest in AI and software. He's brought up the topic in the show.

South Park has a large reach. And the creators have demonstrated a willingness to change their views as they acquire new information. (long ago, South Park satirized Climate Change and Al Gore... but then years later they made a whole "apology episode" that presented Climate Change very seriously... and also apologized to Al Gore)

Seriously, give him a try!

[-]Edouard Harris8d70

Correct & timely. There do exist margins where honesty and effectiveness trade off against each other, but today - in 2025, that is - this is no longer one of them. Your SB 1047 friend is quite right to suggest that things were different in 2023, though. The amount of resistance we got behind the scenes in trying to get words like "extinction" and even "AGI" (!!) published in our report (which was mostly written in 2023) was something to behold: back then you could only push the envelope so far before it became counterproductive. No longer. The best me... (read more)

4Edouard Harris8d

Credit where credit is due, incidentally: the biggest single inflection point for this phenomenon was clearly Situational Awareness. Almost zero reporting in the mainstream news; yet by the end of 2024, everyone in the relevant spaces had read & absorbed it.

[-]TAG19d6-7

Nobel laureates and lab heads and the most cited researchers in the field are saying there’s a real issue here

Some do, others dont. A lot of people are smart enough to spot cherry picking.

[-]dr_s19d119

I think the very fact that some do is already evidence that this is not a completely unfounded concern, which is what this is driving at.

The amount of respected professional doctors who worry about the impact of witches casting curses on public health is approximately zero. The amount of climate scientists who worry about the impact of CO2 emissions on the future of life and civilization on Earth is approximately 100%. The amount of world class AI-related scientists and researchers who worry about AI apocalypse is not as much but it's also far from negligible. If there's a mistake in their thinking, it's the kind of mistake that even a high level expert can make, and it's not particularly more likely than the other side being the one who makes a mistake.

[-]MalcolmMcLeod19d1010

Yes---but when some people say "I think there is danger here" and others say "I think there is no danger here," most people (reasonably!) resolve that to "huh, there could be some danger here"... and the possibility of danger is a danger.

[-]Sherrinford16d40

Thank you for writing this statement on communication strategy and also for writing this book. Even without knowing the specific content in detail, I consider such a book to be very important.

Some time ago, it seemed to me that relevant parts of the AI risk community were ignoring the need for many other people to understand their concerns, instead thinking they could "simply" create a superintelligent and quasi-omnipotent AI that would then save the world before someone else invents paperclip maximizers. This seemed to presuppose a specific worldview th... (read more)

[-]Philip Niewold8d3-2

I don't think people in general react well to societal existential risks, regardless how well or courageous the message is framed. These are abstract concerns. The fact that we are talking about AI (an abstract thing in itself) makes it even worse.

I'm also a very big opponent of arguing by authority (I really don't care how many nobel laureates are of the opinion of something, it is the content of their argument I care about, now how many authorities are saying it). That is simply that I cannot determine the motives of these authorities and hence their opi... (read more)

[-]d_el_ez14d30

Your take is consistent with political messaging advice that people like water and they like drinks but they don't like watered down drinks. Swing voters react to what's said most often, not to the average of things that get said around them.

3Raemon14d

I didn't understand the point you were trying to make here.

[-]rain8dome97d10

Why Mark Ruffalo? Will there be an audiobook? Edit: Yes; it can be preordered now.

[+]ghostintheink8d-26-6

[+][comment deleted]19d175

^{^}

The "courage" property I'm talking advocating here is about naming what you think is the real problem plainly, not about being impolite. This notion of courage is completely compatible with saying "you don't have to agree with me, Mr. Senator, but my best understanding of the evidence is [belief]. If ever you're interested in discussing the reasons in detail, I'd be happy to. And until then, we can work together in areas where our interests overlap." The concept is about naming what you think are the key issues, straightforwardly, in plain language. There are loads of ways to do so politely. See also this comment.

^{^}

That wasn't the only case where the evidence was tricky to read. In January, Senator Cruz had Elon Musk on his podcast and asked “How real is the prospect of killer robots annihilating humanity?” in a context where the senator was overall a bit dismissive. Some of my friends read this as a bad sign, on account of the dismissiveness. I read it it as a good sign, because he was at least starting to think about the key dangers.

And shortly after J.D. Vance read AI 2027, he was asked “Do you think that the U.S. government is capable in a scenario — not like the ultimate Skynet scenario — but just a scenario where A.I. seems to be getting out of control in some way, of taking a pause?” and he answered:

I don’t know. That’s a good question.
The honest answer to that is that I don’t know, because part of this arms race component is if we take a pause, does the People’s Republic of China not take a pause? And then we find ourselves all enslaved to P.R.C.-mediated A.I.?

Some of my friends read this as a bad sign, because he seemed intent on a suicide race. I read it as a good sign, because — again — he was engaging with the real issues at hand, and he wasn't treating AI as just another industry at the mercy of over-eager regulators.

^{^}

My friend caveats that they're still not sure SB 1047 should have been more courageous. All we have actually observed is courage working in 2025, which does not necessarily imply it would have worked in 2023.

^{^}

By "cowardice" here I mean the content, not the tone or demeanor. I acknowledge that perceived arrogance and overconfidence can annoy people in communication, and can cause backlash. For more on what I mean by courageous vs cowardly content, see this comment. I also spell out the argument more explicitly in this thread.

LESSWRONG
LW

494

A case for courage, when speaking of AI danger

494

494