My talk on AI risks at the National Conservatism conference last week

[-]Kaj_Sotala3mo4332

I expected to strong-upvote this because "appealing to conservatives in the kind of language and values that they appreciate and working to join them on issues of AI safety" feels like a very laudable goal. However, much of this talk seemed to be not so much "trying to find common ground and things we can all agree on with the conservatives", instead being "demonizing anyone associated with building AI, including much of the AI safety community itself".

I'm confused how you can simultaneously suggest that this talk is about finding allies and building a coalition together with the conservatives, while also explicitly naming "rationalists" in your list of groups that are trying to destroy religion, among other things. I would expect that the net result of this talk is to makes anyone sympathetic to it discount the opinions of many of the people who've put the most work into understanding e.g. technical AI safety or AI governance.

[-]Wei Dai3mo2314

“rationalists”

Thanks, I had missed this in my reading. It does seem a strange choice to include in the speech (in a negative way) if the goal is to build a broad alliance against building ASI. Many rationalist are against building ASI in our current civilizational state, including Eliezer who started the movement/community.

@geoffreymiller, can you please explain your thought process for including this word in your sentence? I'm really surprised that you seem to consider yourself a rationalist (using "we" in connection with rationalism and arguing against people who do not consider you to be a community member "in good standing"^[1]) and also talk about us in an antagonistic/unfriendly way in front of others, without some overriding reason that I can see.

^{^}
I had upvoted a bunch of your comments in that thread, thinking that we should consider you a member in good standing.

[-]geoffreymiller3mo62

I didn't say that all Rationalists are evil. I do consider myself a Rationalist in many ways, and I've been an active member of LessWrong and EA for years, and have taught several college courses on EA that include Rationalist readings.

What I did say, in relation to my claim that 'they’ve created in a trendy millenarian cult that expects ASIs will fill all their material, social, and spiritual needs', is that 'This is the common denominator among millions of tech bros, AI devs, VCs, Rationalists, and effective accelerationists'.

The 'common denominator' language implies overlap, not total agreement.

And I think there is substantial overlap among these communities -- socially, financially, ethically, geographically.

Many Rationalists have been absolutely central to analyzing AI risks, advocating for AI safety, and fighting the good fight. But many others have gone to work for AI companies, often in 'AI safety' roles that do not actually slow down AI capabilities development. And many have become e/accs or transhumanists who see humanity as a disposable stepping-stone to something better.

[-]Wei Dai3mo179

Yes, on the surface all you did was to point out an overlap between Rationalists and other groups, but what I don't understand is why you chose to emphasize this particular overlap, instead of for example the overlap between us and conservatives of wanting to stop ASI from being built, or simply leaving the Rationalists out of this speech and talk about us another time when you can speak with more nuance.

My hypotheses:

You just want to speak the truth as you see it, without regard to the political consequences. You had room to insert "Rationalist" into that derogatory sentence, but not room to say something longer about how rationalists and conservatives should be allies in this fight.
You had other political considerations that you can't make explicit here, e.g. trying to signal honesty or loyalty to your new potential allies, or preempting a possible attack from other conservatives that you're a Rationalist who shouldn't be trusted (e.g. because we're generally against religion).

I'm leaning strongly towards 2 (as 1 seems implausible given the political nature of the occasion), but still find it quite baffling, in part because it seems like you probably could have found a better way to accomplish what you wanted, without as much of the negative consequences (i.e., alienating the community that originated much of the thinking on AI risk, and making future coalition-building between our communities more difficult).

I think I'll stop here and not pursue this line of questioning/criticism further. Perhaps you have some considerations or difficulties that are hard to talk about and for me to appreciate from afar.

[-]geoffreymiller3mo15-19

Kaj: well, as I argued here, regulation and treaties aren't enough to stop reckless AI development. We need to morally stigmatize anyone associated with building AGI/ASI. That's the main lever of social power that we have.

I see zero prospect for 'technical AI safety work' solving the problem of slowing down reckless AI development. And it's often little more than safety-washing by AI companies to make it look like they take safety seriously - while they continue to push AGI capabilities development as hard as ever.

I think a large proportion of the Rationalist/EA/LessWrong community is very naive about this, and that we're being played by bad actors in the AI industry.

[-]habryka3mo2622

We need to morally stigmatize anyone associated with building AGI/ASI.

No, I think our top priority should be to get people to come to an accurate understanding of the risks associated with AI. I think this requires being able to distinguish between real risks and fake risks. Not everyone associated with AI deserves to be morally stigmatized, and while I agree we should be willing to accept some collateral damage "stigmatizing anyone associated with building AGI" with an implied "by any means necessary^[1]" is IMO not a reasonable strategy.

^{^}
My guess is you do consider some things across the line, but it seems likely to me that the lines of what you consider acceptable to do in the pursuit of stigmatizing people is quite different from me (and my guess is also most other people here)

[-]geoffreymiller3mo130

Ok, let's say we get most of the 8 billion people in the world to 'come to an accurate understanding of the risks associated with AI', such as the high likelihood that ASI would cause human extinction.

Then, what should those people actually do with that knowledge?

Wait for the next election cycle to nudge their political representatives into supporting better AI safety regulations and treaties -- despite the massive lobbying and campaign contributions by AI companies? Sure, that would be nice, and it would eventually help a little bit.

But it won't actually stop AGI/ASI development fast enough or decisively enough to save humanity.

To do that, we need moral stigmatization, right now, of everyone associated with AGI/ASI development.

Note that I'm not calling for violence. Stigmatization isn't violence. It's leveraging human instincts for moral judgment and social ostracism to negate the status and prestige that would otherwise be awarded to people.

If AI devs are making fortunes endangering humanity, and we can't negate their salaries or equity stakes, we can at least undercut the social status and moral prestige of the jobs that they're doing. We do that by calling them our as reckless and evil. This could work very quickly, without having to wait for national regulations or global treaties.

[-]habryka3mo1310

Then, what should those people actually do with that knowledge?

Focus a mixture of stigma, regulation, and financial pressures on the people who are responsible for building AGI/ASI. Importantly "responsible" is very different from "associated with".

If AI devs are making fortunes endangering humanity, and we can't negate their salaries or equity stakes, we can at least undercut the social status and moral prestige of the jobs that they're doing.

Yep, I am in favor of such stigmas for people working on frontier development. I am not in favor of e.g. such a stigma for people who are developing self-driving cars, or are working on stopping AI themselves (and as such are "associated with building AGI/ASI").

I think we both agree pretty strongly that I think there should be a lot of negative social consequences for people responsible for building AGI/ASI. My sense is you want to extend this further beyond "responsible" and into "associated with", and I think this is bad. Yes, we can't expect perfect causal models from the public and the forces behind social pressures, but we can help make them more sane and directed towards the things that help, as opposed to the things that are just collateral damage or actively anti-helpful. That's all I am really asking for.

[-]geoffreymiller3mo142

Oliver -- that's all very reasonable, and I largely agree.

I've got no problem with people developing narrow, domain-specific AI such as self-driving cars, or smarter matchmaking apps, or suchlike.

I wish there were better terms that could split the AI industry into 'those focused on safe, narrow, non-agentic AI' versus 'those trying to build a Sand God'. It's only the latter who need to be highly stigmatized.

Peace out :)

[-]Matrice Jacobine3mo40

Ok, let's say we get most of the 8 billion people in the world to 'come to an accurate understanding of the risks associated with AI', such as the high likelihood that ASI would cause human extinction.
Then, what should those people actually do with that knowledge?

Boycotting any company which fund research into AGI would be (at least in this scenario) both more effective and, for the vast majority of people, more tractable than "ostracizing" people whose social environment is largely dominated by... other AI developers and like-minded SV techno-optimists.

[-]geoffreymiller3mo20

Matrice -- for more on the stigmatization strategy, see my EA Forum post from a couple years ago, here

IMHO, a grassroots moral stigmatization campaign by everyone who knows AGI devs would be much more effective that just current users of a company's products boycotting that company.

[-]Matrice Jacobine3mo1-1

As a reality check, "any company which fund research into AGI" here would mean all the big tech companies (MAGMA). Much more people use those products than people know AGI developers. It is a much easier ask to switch to using a different browser/search engine/operating system, install an ad blocker, etc. than to ask for social ostracism. Those companies' revenues collapsing would end the AI race overnight, while having AGI developers keep a social circle of techno-optimists only wouldn't.

[-]geoffreymiller3mo40

Matrice -- maybe, if it was possible for people to boycott Google/Deepmind, or Microsoft/OpenAI.

But as a practical matter, we can't expect hundreds of millions of people to suddenly switch from gmail to some email alternative, or to switch from Windows to Linux.

It's virtually impossible to organize a successful boycott of all the Big Tech companies that have oligarchic control of people's digital lives, and that are involved in AGI/ASI development.

I still think the key point of leverage of specific, personalized, grassroots social stigmatization of AGI/ASI developers and people closely involved in what they're doing.

(But I could be convinced that Big Tech boycotts might be a useful auxiliary strategy).

[-]Matrice Jacobine3mo*10

Your scenario above was that most of the 8 billion people in the world would come to believe with high likelihood that ASI would cause human extinction. I think it's very reasonable to believe that this would make it quite easier to coordinate to make alternatives to MAGMA products more usable in this world, as network effects and economies of scale are largely the bottleneck here.

[-]Kaj_Sotala3mo1711

We need to morally stigmatize anyone associated with building AGI/ASI

This sounds like a potentially sensible strategy to me. But you say yourself in the intro that you don't want AI to become a partisan issue, and then... go on to speak about it in highly partisan terms, as a thing pushed by Those Evil Liberals?

If you are talking about AI and listing various ways that AI could be used against conservatives, what's to prevent various liberals from going "hey good idea, let's build AI to shaft conservatives in exactly the way Geoffrey is describing"? Or from just generally going "aha conservatives seem to think of AI as a particularly liberal issue, I hadn't thought of my stance on AI before but if conservatives hate it as a leftist globalist issue then I should probably support it, as a leftist globalist myself?"

[-]geoffreymiller3mo1-2

Kaj -- I think the key thing here is to try to avoid making AI safety a strongly partisan-coded issue (e.g. 'it's a Lefty thing' or 'its a Righty thing') -- but to find persuasive arguments that appeal about equally strongly to people coming from different specific political and religious values.

So, for example, conservatives on average are more concerned about 'the dignity of work' and 'the sanctity of marriage', so I emphasized how ASI would undermine both (through mass unemployment and AI sexbots/waifus/deepfakes), because that's how to get their attention and interest. Whereas liberals on average may be more concerned about 'economic inequality', so when speaking with them, it might be more effective to talk about how ASI could dramatically increase wealth differences between future AI trillionaires and ordinary unemployed people.

So it's really about learning specific ways to appeal to different constituencies, given the values and concerns they already have -- rather than making AI into a generally liberal or generally conservative cause. Hope that makes sense.

[-]Kaj_Sotala3mo129

So, for example, conservatives on average are more concerned about 'the dignity of work' and 'the sanctity of marriage', so I emphasized how ASI would undermine both (through mass unemployment and AI sexbots/waifus/deepfakes), because that's how to get their attention and interest.

I agree that this is a sensible strategy to pursue, and it's why I originally expected to strong-upvote your post since I thought that this is exactly something that we need to do.

But you could do that in a way that emphasizes commonalities and focuses on the concrete risks rather than using partisan language. For instance, when you say

the big AI companies wants its users to develop their most significant, intimate relationships with their AIs, not with other humans

... then this doesn't really even feel true? Okay, Grok's companion feature is undeniably sketchy, but other than that, I haven't heard any AI developers saying or hinting that they would want their users to develop their most significant relationships with AIs rather than humans. There are a few explicit companion apps, but those largely seem to be oriented at complementing rather than replacing human relationships.

If anything, many of the actions of "the big AI companies" seem to indicate the opposite. OpenAI seemed to be caught off balance when they made GPT-5 significantly less sycophantic and emotionally expressive, as if they hadn't even realized or considered that their users might have been developing emotional bonds with 4o that the update would disrupt. And I recall seeing screenshots where Claude explicitly reminds the user that it's a chatbot and it would be inappropriate for it to take on the role of a loved human. ChatGPT, Claude, and Gemini are all trained to refuse sexual role-play or writing and while those safeguards are imperfect, the companies training them do seem to consider that a similar level priority as preventing them from producing any other kind of content deemed harmful.

So this quote from your talk doesn't read to me as "truthfully emphasizing the way that AI might undermine sanctity of marriage", it reads to me more as "telling a narrative of how AI might undermine the sanctity of marriage by tapping into the worst strawmen stereotypes that conservatives have about liberals". (With a possible exception for X/Grok. I could accept this description as at least somewhat truthful if you were talking about just them, rather than "the big AI companies" in general.)

And it feels to me totally unnecessary, since you could just as well have said something like

"You wouldn't usually think of sanctity of marriage as a thing that liberals care about, but there have been such worrying developments with regard to people interacting with chatbots that even they are concerned about the effect that AI has on relationships. They have expressed concerns about the way that some people develop deep delusional relationships with sycophantic AIs that tell the users exactly what the users want to hear, and how people might start preferring this over the trickiness and messiness of real human relationships. Can you imagine how worrying things are getting if even liberals think that technology might be intruding too much on something intimate and sacred? We disagree on a lot of things, but on this issue, they are with us."

[-]StanislavKrym3mo60

Grok's companion feature is undeniably sketchy, but other than that, I haven't heard any AI developers saying or hinting that they would want their users to develop their most significant relationships with AIs rather than humans.

Except that there also is Meta with Zuckerberg's AI vision. Fortunately, OpenAI, Anthropic and GDM didn't hint on planning to create AI companions.

@geoffreymiller: as for AIs destroying marriage, I see a different mechanism for this. Nick Bostrom's Deep Utopia has the AIs destroy all instrumental goals that humans once had. This arguably means that humans have no way to help each other with anything or to teach anyone anything, they can only somehow form bonds over something and will even have a hard time expressing those bonds. And that's ignoring the possibilities like the one proposed in this scenario:

Many ambitious young women see socialising as the only way to wealth and status; if they start without the backing of a prominent family or peer group, this often means sex work (sic! -- S.K.) pandering to spoiled millionaires and billionaires.

[-]Kaj_Sotala3mo50

Thanks for the link, I hadn't followed what Zuckerberg is up to and also wasn't counting Meta as one of the major AI players, though probably I should. Apparently, he does also claim that he intends AIs to be complements rather than substitutes, but I do agree with Zvi that his revealed preferences with promoting AI-generated content over human content point the other way. So I guess there's two big AI companies that geoffrey's characterization kinda applies to.

[-]dr_s3mo50

With a possible exception for X/Grok

Which was explicitly created by Musk to be the anti-woke AI so I really don't see how it would connect to liberals.

[-]Remmelt3mo31

instead being "demonizing anyone associated with building AI, including much of the AI safety community itself".
I'm confused how you can simultaneously suggest that this talk is about finding allies and building a coalition together with the conservatives, while also explicitly naming "rationalists" in your list of groups that are trying to destroy religion

I get the concern about "rationalists" being mentioned. It is true that many (but not all) rationalists tend to downplay the value of traditional religion, and that a minority of rationalists unfortunately have worked on AI development (including at DeepMind, OpenAI and Anthropic).

However, I don't get the impression that this piece is demonising the AI Safety community. It is very much arguing for concepts like AI extinction risk that came out of the AI Safety community. This is setting a base for AI Safety researchers (like Nate Soares) to talk with conservatives.

The piece is mostly focussed on demonising current attempts to develop 'ASI'. I think accelerating AI development is evil in the sense of 'discontinuing life'. A culture that commits to not do 'evil' also seems more robust at preventing some bad thing from happening than a culture focused on trying to prevent an estimated risk but weighing this up with estimated benefits. Though I can see how a call to prevent 'evil' can result in a movement causing other harms. This would need to be channeled with care.

Personally, I think it's also important to build bridges across to multiple communities, to show where all of us actually care about restricting the same reckless activities (toward the development and release of models). A lot of that does not require bringing up abstract notions like 'ASI', which are hard to act on and easy to conflate. Rather, it requires relating with communities' perspectives on what company activities they are concerned about (e.g. mass surveillance and the construction of hyperscale data centers in rural towns), in a way that enables robust action to curb those activities. The 'building multiple bridges' aspect is missing in Geoffrey's talk, but also it seems focused on first making the case why traditional conservatives should even care about this issue.

If we care to actually reduce the risk, let's focus the discussion on what this talk is advocating for, and whether or not that helps people in communities orient to reduce the risk.

[-]Three-Monkey Mind3mo94

while also explicitly naming “rationalists” in your list of groups that are trying to destroy religion

Quite honestly, if he’s not mentioning things like that, then the rest of what he says comes across as lying by omission to people who are sick and tired of being lied to by omission.

I would expect that the net result of this talk is to makes anyone sympathetic to it discount the opinions of many of the people who’ve put the most work into understanding e.g. technical AI safety or AI governance.

My prior is that most of what is called “AI safety” work is Timnit compliance, not notkilleveryoneist work, but I’m open to updating.

[-]Kaj_Sotala3mo52

Quite honestly, if he’s not mentioning things like that, then the rest of what he says comes across as lying by omission to people who are sick and tired of being lied to by omission.

It's of course true that rationalists tend to be hostile to religion, and that this would be untruthful to deny.

But you only have a limited amount of things you can say in a talk. If you bring up something, it's because you want to explicitly emphasize and draw attention to that. Everything else gets omitted automatically, and there's nothing dishonest about that. Most of the audience isn't even going to know who rationalists are, I don't think that anyone would have reacted to the speech negatively if that part had been left out - they'd just have focused on the things that geoffrey did say that they agreed with.

And if one did want to bring up the religion angle, one could have done it in a way that emphasized the agreement. "Even many liberals are now afraid of humanity creating a faux god that they cannot control. While they do not think of it in those terms, even they can see that the mania of the most reckless AI developers is setting up a new Tower of Babel that may shatter not only language but the world itself."

[-]geoffreymiller3mo40

Yes. And way too much 'AI safety work' boils down to 'getting paid huge amounts by AI companies to do safety-washing & public relations, to kinda sorta help save humanity, but without upsetting my Bay Area roommates & friends & lovers who work on AI capabilities development'.

[-]RussellThor3mo91

Understandable position well articulated.
An important issue I have with conservatism (and many AI safety positions) is that it assumes the kind of world that arguably doesn't exist. That is one that is somewhat stable, safe, and sustainable. The assumption is that without AI the good things we currently know would continue for long enough to matter.
If instead we view ourselves in an unlikely timeline where the most likely outcome from the last 100 years is that we have had a full on nuke war then that changes the perspective. Considering all the close calls, if there in hindsight was a 75% chance of nuke war from 1960 till now and we are just lucky then that changes much.

Given that such odds probably havn't changed i.e. great power conflict with China taking the place of Russia in the next 75 years will give similar dangers, then our current situation is not one to preserve, but instead change as soon as we can. You talk about Russian Roulette but perhaps want to preserve a situation where we arguably already play it every 100 years with 5 bullets in the chamber. That is not including new threats - does pre LLM AI/social media cause collapse after time? Does LLM + dictators cause a permanent 1984 style world given time?

If you believe that humanity is an a very dangerous phase of unavoidable change then it is about getting out of that situation with the highest chance, rather then attempting to preserve the current seemingly safe-ish situation. ASI is one way, large scale space colonization (different stars) is another.

[-]geoffreymiller3mo32

Russell -- I take your point that in most alternative timelines, we would already be dead, decades ago, due to nuclear war, and I often make that point in discussing AI risk, to hammer home the point that humanity does not have any magical 'character armor' that will protect us from extinction, and that nobody is coming to save us if we're dumb enough to develop AGI/ASI.

However, I disagree with the claim that 'our current situation is not one to preserve'. I know people in the military/intelligence communities who work full time on nuclear safety, nuclear non-proliferation, counter-terrorism, etc. There are tens of thousands of smart people across dozens of agencies across many countries who spend their entire lives reducing the risks of nuclear war. They're not just activists making noise from outside the centers of power. They're inside the government, with high security clearances, respected expertise, and real influence. I'm not saying the risk of nuclear war has gone to zero, but it is taken very seriously by all the major world governments.

By contrast, AI safety remains something of a fringe issue, with virtually no representation inside governments, corporations, media, academia, or any other power centers. That's the thing that needs to change.

We don't need a 'hail Mary' where we develop AGI/ASI and then hope that it can reduce nuclear risk more than it increases all other risks.

[-]RussellThor3mo*10

I am glad there are people working on nuclear safety and absolutely agree there should be more AI safety inside governments.
I also think pre-LLM AI tech should get more attention - Peter Thiel I think makes the point that software has very little regulation compared to much physical things yet it can have enormous influence. I'm sure I don't need to persuade you that the current dating situation is not ideal. What can be practically done about it all things considered however is not so clear.

However those nuke safety people aren't working inside Russia as far as I am aware? My point is that we still don't know what such risk is as of now, nor do we have much of an estimate in the coming decades. The justifiable uncertainty is huge. My position when considering a pause/stop depends on weighing up things we can really only guess at.

To consider when say delaying ASI 50+ years we need to know:

What is the chance of nuke war/lethal pandemic etc in that time? 2%, 90%?

What will LLM tech and similar do to our society?
Specifically what is the chance that it will degrade our society in some way that when we do choose to go ahead with ASI we get "imagine a boot stamping on a human face – for ever." While pure x-risk may be higher with immediate ASI, I think S-risk will be higher with a delay. In the past, centralization and dictators would eventually fail. Now imagine if a place like N Korea gave everyone a permanent bracelet that recorded everything they said paired to an LLM that also understood their hand gestures and facial expressions. They additionally let pre-ASI AI run their society so that central planning actually could work. I think that there is no coming back from that.

Now even if such a country is economically weaker than a free one, if there is a % chance each decade that free societies fall into such an attractor, then eventually the majority of economic output ends up in such a system. They then "solve" alignment getting an ASI that does their bidding.

What is the current X-risk, and what would it be after 50 years of alignment research?

I believe that pre GPT-3/3.5, further time spent on alignment would be essentially a dead end. Without actual data we get into diminishing returns, and likely false confidence on results and paradigms. However it is clear that X-risk could be a lot lower if done right. To me that means actually building ever more powerful AI in very limited and controlled situations. So yes a well managed delay could greatly reduce X-risk.

There are 4 very important unknowns here, potentially 5 if you separate out S risk. How to decide? Is +2% more S-risk acceptable if you take X risk from 50% to 5%? Different numbers for these situations will give very different recommendations. If the current world was going well, then sure its easy to see that a pause/stop is the best option.

What to do?

From this it is clear that work on actually making the current world safer is very valuable. That is protecting institutions that work, anticipating future threats and making the world more robust against them. Unfortunately that doesn't mean that keeping the current situation as long as possible is the best all things considered.

If someone thought there is a high chance that ASI is coming soon or that even with the best efforts the current world can't be made sufficiently safe, then they would want to work on making ASI go well, for example mechanistic interpretability research or other practical alignment work.

Expressing such uncertainty on my part probably won't get me invited to make speeches and can come across as a lack of moral clarity. However it is my position and I don't think behavior based on the outcome of those uncertainties should be up for moral stigmatization.

These are not my numbers but lets say you have 50% for nuke war/similar event, then 50% for S-risk from the surviving worlds over the next 100 years with no ASI, but 20% X risk/1% S risk from ASI < 5 years. Your actions and priorities are then clear and morally defensible from your probabilities. Some e/acc people may genuinely have these beliefs.

Edited later for my reference
Does pursuing WBE change this? Perhaps if you think we can delay ASI but just 20 years to get WBE and believe that they will be better aligned. If you get ASI first and then use them to create WBE that can be seen perhaps as a pivotal act. Stop pure AI but only create WBE is not a strategy I have seen pushed seriously. It doesn't seem possible without first having massive GPU control etc as its pretty clear without constraints pure AI will be made first. For example if you have the tech to scan enough of a brain, then you are pretty much guaranteed to be able to make ASI from what you have learnt before you have scanned the whole brain.

[-]Seth Herd3mo53

I've been excited about pitching AGI x-risks to conservatives since seeing the great outreach work and writup from AE Studios, Making a conservative case for alignment.

My fervant hope is that we somehow avoid making this a politically polarized issue. I fear that polarization easily overwhelms reason, and is one of the few ways the public could fail to appreciate the dreadful, simple logic in time to be of any help.

[-]geoffreymiller3mo43

Seth - thanks for sharing that link; I hadn't seen it, and I'll read it.

I agree that we should avoid making AI safety either liberal-coded or conservative-coded.

But, we should not hesitate to use different messaging, emphasis, talking points, and verbal styles when addressing liberal or conservative audiences. That's just good persuasion strategy, and it can be done with epistemic and ethical integrity.